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ABSTRACT 


The  aim  of  this  thesis  is  to  develop  and  test  a  proof-of-coneept  augmented  reality  display 
that  presents  critieal  navigation  information  to  naval  eonning  officers.  The  objective  of 
this  research  effort  was  to  study  the  feasibility  and  usability  of  such  an  approach  in 
operational  conditions.  The  testbed  platform  consisted  of  a  virtual  environment  that  fully 
simulated  a  conning  officer’s  basic  tasks  in  conditions  of  restricted  navigation;  this  type 
of  setup  enabled  a  cost-effective  test  solution  that  was  safe  and  supported  scenario 
repeatability  in  studies  with  human  subjects.  The  study  involved  25  experienced  test 
subjects  who  were  surface  warfare  officers  at  both  the  Naval  Postgraduate  School  and 
Surface  Warfare  Officer  School.  This  effort  helped  acquire  a  comprehensive  set  of 
objective  and  subjective  data  that  provided  a  close  insight  into  the  performance  of 
conning  officers.  The  empirical  results  demonstrate  the  viability  of  using  such  a  system  in 
an  operation  environment  and  support  a  need  for  further  research  and  development  of  a 
working  display  platform  onboard  Navy  warships. 
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I.  INTRODUCTION 


A.  RESEARCH  DOMAIN 

Despite  the  introduetion  of  Global  Positioning  System  (GPS)  and  eleetronie 
navigation,  the  art  and  seience  of  eonning  a  naval  vessel  has  ehanged  very  little  over  the 
past  half-eentury.  Driving  military  ships  is  still  a  heavily  manual  praetice  and,  since  1980, 
there  have  been  over  100  navigation  related  mishaps  significant  enough  to  warrant  formal 
investigation  (Department  of  the  Navy  Naval  Safety  Center,  2014).  As  an  example,  in 
February  of  2013,  USS  Guardian  (a  minesweeper  forward  deployed  to  Japan)  ran 
aground  in  the  Philippines.  This  incident  not  only  caused  irreparable  damages  to  the  local 
reef  system,  but  resulted  in  the  complete  loss  of  the  vessel  (U.S.  Pacific  Fleet  Public 
Affairs,  2013).  In  July  2000,  a  collision  at  sea  between  USS  Denver  and  USNS  Yukon 
resulted  in  the  former  receiving  “a  gaping  40-foot  hole  in  the  bow  from  the  second  deck 
to  the  waterline”  (Doehring,  2014).  In  September  of  the  same  year,  USS  La  Moure 
County  ran  aground  off  the  coast  of  Chile,  receiving  damages  significant  enough  to 
warrant  decommissioning  and  later  scuttling  (Doehring,  2012).  While  there  are  always 
numerous  contributing  factors  to  groundings  and  collisions,  the  fact  that  these  incidents 
continue  to  occur  clearly  indicates  a  need  for  more  effective  solutions  and  tools  to  assist 
navigation  tasks. 

If  the  U.S.  Navy  continues  the  historical  trend  of  having  its  officers  manually 
drive  ships,  then  every  effort  to  augment  human  capabilities  and  incorporate  current 
sensor  technologies  into  the  decision-making  loop  must  be  made.  The  electronic  sensors 
on  ships  already  collect  and  process  enough  data  to  make  safer  navigation  a  reality.  It  is 
an  efficient  presentation  of  that  information  to  the  human  operator  that  is  lacking;  a  real¬ 
time  visual  display  system  always  accessible  to  conning  officers  and  in  their  visual  field 
of  view  (while  still  allowing  them  to  focus  on  the  visuals  of  the  real  world  in  front  of 
their  eyes)  has  yet  to  be  developed.  If  such  a  system  is  to  be  adopted  on  future  ships  and 
serve  the  needs  of  conning  officers,  it  must  be;  relatively  cheap,  absolutely  reliable, 
robust,  rugged,  and  highly  effective  at  relaying  the  navigational  picture  to  the  human 
operator. 
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B, 


MOTIVATION  FOR  RESEARCH 


(1)  Navigation  Mishaps  are  Costly 

Since  1980,  over  8  percent  of  the  total  number  of  Class  A  and  Class  B  mishaps 
that  occurred  in  the  Navy  have  been  navigation  related  (Department  of  the  Navy  Naval 
Safety  Center,  2014).  The  Office  of  the  Chief  of  Naval  Operations  defines  Class  A  and 
Class  B  mishaps  as  follows  (Office  of  the  Chief  of  Naval  Operations  &  Commandant  of 
the  Marine  Corps,  2005,  p.  2-2) 

Class  A  Mishap.  The  resulting  total  cost  of  damages  to  DOD  or  non-DOD 
property  in  an  amount  of  $2  million  or  more;  a  DOD  aircraft  is  destroyed; 
or  an  injury  and/or  occupational  illness  result  in  a  fatality  or  permanent 
total  disability. 

Class  B  Mishap.  The  resulting  total  cost  of  damages  to  DOD  or  non-DOD 
property  is  $500,000  or  more,  but  less  than  $2  million.  An  injury  and/or 
occupational  illness  result  in  permanent  partial  disability  or  when  three  or 
more  personnel  are  hospitalized  for  inpatient  care  (beyond  observation)  as 
a  result  of  a  single  mishap. 

Although  the  incidents  only  average  to  three  per  year,  each  accident  is  a 
significant  event  with  strategic  consequences  (Department  of  the  Navy  Naval  Safety 
Center,  2014).  In  the  case  of  USS  Guardian,  (one  of  only  a  few  remaining  mine- 
countermeasure  ships)  the  U.S.  Navy  lost  a  significant  percentage  of  its  capability  to  hunt 
and  disable  mines.  With  each  new  naval  platform  being  more  robust  and  more  costly,  the 
loss  of  only  a  few  ships  to  causes  such  as  preventable  navigation  mishaps  cannot  be 
afforded. 

(2)  Current  Practices  are  Error  Prone 

Methodology  of  how  ships  are  driven  in  today’s  environments  is  detailed  in 
Chapter  III.  In  general,  the  navigation  evaluator  gives  directions  to  the  conning  officer, 
who  then  orders  the  helmsman  to  manipulate  the  ship’s  rudder  and  engines.  This 
communication  path  requires  constant  alertness  of  all  parties  and  complete  focus  on  the 
task.  If  communications  are  misunderstood  or  misinterpreted,  the  ship  can  be  driven  into 
dangerous  waters  that  put  the  entire  crew  at  risk.  Minor  variables  such  as  personal 
accents,  vocal  inflections,  or  sickness  can  negatively  impact  the  effectiveness  of  these 
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communications.  External  forces  sueh  as  maehinery  noise,  heavy  rain,  or  adjacent  traffie 
ean  also  induee  errors  in  understanding  the  reports  or  orders.  The  helmsman  is  trained  to 
repeat  baek  his  orders  to  the  eonning  offieer  before  execution,  with  the  goal  of  mitigating 
some  of  these  issues;  however,  the  reports  and  recommendations  from  the  navigation 
evaluator  are  not  repeated.  Given  all  these  issues,  the  overall  proeedure  for  navigation 
through  restrieted  waters  ean  and  should  be  improved. 

C.  RESEARCH  QUESTIONS 

The  following  researeh  questions  are  the  foeal  points  in  this  thesis: 

•  Does  replaeing  the  auditory  inputs  from  the  navigation  officer  with  an 
augmented  field  of  view  (FOV)  that  visually  feeds  eritieal  navigation 
information  (CNI)  to  the  eonning  offieer  aid  or  hinder  his  performanee? 

•  If  a  heads-up  display  (HUD)  system  was  designed  and  deployed  to  the 
fleet,  would  surfaee  warfare  offieers  be  reeeptive  to  the  idea  of  integrating 
and  using  sueh  a  system  in  their  daily  operations? 

D,  HYPOTHESES 

For  the  purpose  of  this  thesis  research,  the  following  two  null  hypotheses  and 
alternative  hypotheses  have  been  established: 

(1)  Hypothesis  1 

•  Null  Hypothesis:  HIq — A  lightweight,  glasses-type  augmented  reality 
(AR)  overlay  system  for  a  eonning  officer  does  not  inerease  his  ability  to 
maintain  a  elose  proximity  to  a  preplanned  traek. 

•  Alternative  Hypothesis:  HI  a — A  lightweight,  glasses-type  AR  overlay 

system  for  a  eonning  offieer  will  inerease  his  ability  to  maintain  a  close 
proximity  to  a  preplanned  track. 

(2)  Hypothesis  2 

•  Null  Hypothesis:  H2o — The  surface  warfare  offieers  will  not  be  reeeptive 

to  the  idea  of  integrating  and  using  sueh  a  system  into  their  daily 
operations 

•  Alternate  Hypothesis:  H2a — The  surfaee  warfare  offieers  will  be  receptive 
to  the  idea  of  integrating  and  using  sueh  a  system  into  their  daily 
operations 
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E,  SCOPE 

The  scope  of  this  thesis  will  be  to  study  the  effectiveness  of  an  AR  overlay  for 
conning  officers  in  the  situation  of  an  outbound  channel  transit  with  no  traffic  or 
navigation  aids  in  the  adjacent  water.  The  purpose  of  not  including  surface  vessel  traffic 
in  the  simulation  was  to  reduce  the  number  of  variables  in  the  experiment.  Just  as  there 
are  numerous  methods  to  driving  ships,  so  too  are  there  methods  to  dealing  with 
oncoming  surface  traffic.  While  the  best  way  to  test  such  a  system  would  be  going  to  a 
physical  ship  with  a  commercial  off-the-shelf  hardware  prototype,  there  are  numerous 
safety  and  repeatability  concerns  that  make  such  testing  unrealistic.  Therefore,  our 
decision  was  to  conduct  all  tests  in  a  fully  immersive  three-dimensional  (3D)  virtual 
shipboard  environment  via  a  head-mounted  display  (HMD). 

The  computer  supported  simulation  will  allow  us  to:  present  the  same  system 
conditions  and  scenario  to  a  human  operator,  capture  a  precise  objective  data  set  related 
to  the  operator’s  performance,  and  to  complete  that  test  with  utmost  safety  provided  to 
the  subjects.  Virtual  environments  have  been  proven  as  effective  testbeds  capable  of 
simulating  and  studying  the  real  shipboard  operations;  this  in  turn  suggests  that  the 
results  acquired  in  this  study  will  indicate  the  system’s  applicability  and  relevance  to  the 
real-world  situations  (de  Moraes,  2011). 

F.  THESIS  CONTRIBUTIONS 

This  thesis  will  provide  empirical  data  on  the  usefulness  of  AR  concepts  for 
conning  officers.  Questionnaires  and  interviews  were  used  to  capture  study  subjects’ 
opinions  on  the  operational  usefulness  of  such  a  system.  These  opinions  come  from 
experienced  officers  who  represent  the  community  of  shareholders  who  may  end  up  using 
this  technology  in  the  near  future.  The  technical  discussion  of  how  to  best  design  the 
layout  for  the  augmented  layer  is  also  provided.  Creating  a  HUD  that  was  able  to  provide 
ample  information  while  not  being  overly  distracting  was  a  critical  goal  of  the  study.  The 
study  design  balanced  all  experimental  and  domain  requirements,  and  the  final  results 
were  documented  after  conducting  analysis  of  tracking  data  and  questionnaires. 
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G.  THESIS  STRUCTURE 

Chapter  I  is  an  introduction  of  the  overall  goals  and  methods  of  the  study. 

Chapter  II  details  a  collection  of  background  information  covering  both  the 
doctrine  and  the  technology  employed  onboard  today’s  ships.  Additionally,  this  section 
contains  a  literature  review  on  the  topics  and  issues  related  to  VR  and  AR  systems. 

Chapter  III  presents  a  description  of  daily  responsibilities  of  those  in  charge  of  the 
navigation  of  a  ship. 

Chapter  IV  details  the  environment  used  for  the  user  study.  This  covers  both  the 
technical  hardware  descriptions  as  well  as  describing  the  process  of  creating  the  virtual 
testing  environment. 

Chapter  V  covers  the  entirety  of  the  usability  study  and  data  analysis. 

Chapter  VI  offers  a  conclusion  and  several  insights  into  recommendations  for 
future  work. 
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II.  BACKGROUND 


This  chapter  provides  background  knowledge  of  the  operational,  teehnological, 
and  physiological  models  directly  related  to  the  thesis  research  domain  and  consequently 
used  in  the  development  and  testing  of  the  Navigation  Heads-up  Display  (NAVHUD) 
system.  The  ehapter  covers  the  doctrinal  policies  of  U.S.  Navy  restrieted  maneuvering 
operations.  Additionally,  the  section  provides  a  literature  review  of  the  VR  and  AR  fields. 
Finally,  the  physiologieal  phenomena  assoeiated  with  virtual  environments  (VE)  are 
discussed.  This  ehapter  provides  only  a  rudimentary  evaluation  of  each  of  these  domains, 
offering  enough  information  for  the  reader  to  comprehend  the  experimental  design  and 
exeeution. 

A,  RESTRICTED  MANUEVERING 

The  starting  goal  for  the  study  was  to  limit  testing  of  the  NAVHUD  system  to 
conditions  that  made  sense  operationally,  and  thusly  a  decision  to  only  cover  restricted 
maneuvering  situations  was  adopted.  Restricted  maneuvering  is  an  operational  term  used 
for  situations  when  the  ship  is  restricted  in  its  ability  to  maneuver  either  due  to 
environmental  or  operational  reasons.  The  seminal  text  for  newly  reporting  surface 
warfare  offieers  defines  four  specific  conditions  for  which  the  ship  is  eonsidered  in 
restricted  maneuvering  (Stavridis  &  Girder,  2007,  p.  329): 

1 .  Operating  in  restrieted  waters 

2.  Steaming  in  elose  formation 

3 .  Conducting  an  underway  replenishment 

4.  Engineering  easualties  affeeting  the  ship’s  ability  to  maneuver 

The  focus  of  this  work  is  the  most  common  of  these  situations — operations  in 
restricted  waters.  Several  actions  must  be  taken  by  the  ship’s  erew  for  any  of  the  four 
conditions.  Eirstly,  several  additional  bridge  and  engineering  watehes  must  be  stood  up. 
This  is  intended  to  not  only  provide  greater  accuracy  of  maneuvering  but  to  also  provide 
an  immediate  reaction  capability  in  the  event  of  any  casualties.  Additionally,  the 
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engineering  plant  is  required  to  be  eonfigured  in  a  maximum  reliability  lineup  (Stavridis 
&  Girrier,  2007,  p.  329).  Although  the  aetual  definition  of  maximum  reliability  differs  for 
eaeh  ship,  this  proeess  normally  involves  bringing  all  the  main  engines  online,  preparing 
supplemental  generators  for  immediate  use,  testing  the  after  steering  controls,  and 
manning  additional  engineering  watches.  The  purpose  of  these  acts  is  to  provide 
maximum  capability  to  individuals  driving  the  ship  from  the  bridge. 

(1)  Bridge  Team 

When  steaming  in  open  ocean  (more  than  10  nautical  miles  from  land)  and  not 
conducting  operations,  the  bridge  of  a  U.S.  warship  can  have  as  few  as  three  personnel  on 
watch:  the  officer  of  the  deck  (OOD),  acting  as  primary  supervisor  and  doubling  as  a 
conning  officer;  the  helmsman  controlling  the  ship;  and  a  quartermaster,  acting  as  the 
navigation  department  representative.  This  watch  configuration  changes  dramatically 
when  conducting  restricted  maneuvering  operations.  With  as  many  as  20  personnel 
standing  critical  bridge  watch  positions,  only  the  roles  of  those  who  are  primarily  tasked 
with  the  safe  navigation  of  the  ship  will  be  detailed. 

•  OOD:  The  primary  supervisor  for  all  watch  stations,  the  OOD  is  the 
captain’s  direct  representative  on  the  bridge.  Tasked  with  the  safe 
execution  of  naval  operations,  the  OOD  is  overall  responsible  for  the 
safety  of  the  ship  and  crew. 

•  Conning  Officer:  The  conning  officer  is  tasked  with  the  physical 
maneuvering  of  the  ship.  By  issuing  engine  and  rudder  orders  to  the 
helmsman,  the  conning  officer  drives  the  ship.  He  combines  the 
recommendations  provided  by  the  navigation  evaluator,  and  the  current 
surface  contact  picture,  to  keep  the  ship  safe  from  collisions  and 
groundings. 

•  Helmsman:  The  helmsman  takes  the  orders  of  the  conning  officer  and 
turns  them  into  physical  actions.  The  helmsman  does  not  make 
independent  decisions  on  when  to  turn  or  what  course  to  come  to.  Rather, 
he  provides  a  logical  safety  barrier  for  situations  where  the  conning  officer 
may  give  an  incorrect  order. 

•  Navigation  Evaluator:  The  navigation  evaluator  is  responsible  for 
reporting  where  the  ship  is  currently  positioned.  Comparing  several 
sources  available  to  him,  the  navigation  evaluator  can  make  a  precise 
determination  of  the  ship’s  position.  With  this  information,  the  navigation 
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evaluator  recommends  courses  to  the  conning  officer  that  will  keep  the 
ship  in  safe  water. 

(2)  Restricted  Waters 

Defined  as  waters  that  are  less  than  two  nautical  miles  from  land  or  shoal  water, 
restricted  waters  are  transited  on  a  daily  basis  by  U.S.  Navy  warships 
(COMNAVSURFPAC  et  al.,  2013).  While  the  most  common  transits  of  restricted  waters 
are  entering  and  exiting  ports,  there  are  many  other  cases  where  a  vessel  might  be 
required  to  come  within  two  miles  from  land.  These  include  coastal  patrols,  strait  transits, 
minesweeping  operations,  or  even  rendering  assistance  to  distressed  mariners.  In  all  cases 
where  the  ship  enters  restricted  waters,  the  ship  is  required  to  stand  up  the  watches  and 
engineering  plant  configurations  associated  with  restricted  maneuvering  operations. 

B,  CURRENT  TECHNOLOGICAL  ENVIRONMENT 

The  U.S.  Navy  has  invested  heavily  in  the  research  and  development  of 
technologies  that  aid  in  the  safe  navigation  of  its  ships.  The  following  systems  are  used 
on  a  daily  basis  to  track  surface  contacts,  communicate  with  merchant  vessels,  and  plan 
navigation  transits. 

(1)  Radar 

There  are  numerous  commercial  and  military  radar  systems  installed  on  today’s 
Navy  ships.  These  include  navigation  radars,  surface  search  radars,  and  air  control  radars 
(U.S.  Navy,  n.d.).  These  systems  give  the  U.S.  Navy  the  capability  to  create  an 
operational  picture  of  what  vessels  and  aircraft  are  maneuvering  in  the  area.  Using  this 
information,  the  ship’s  crew  can  prevent  collisions  by  anticipating  the  movements  of 
nearby  traffic. 

Although  primarily  associated  with  the  discovery  and  classification  of  air  and 
surface  contacts,  radar  is  also  a  useful  mechanism  for  indicating  where  the  land  is.  The 
radio  waves  bounce  off  of  any  surface  that  protrudes  out  of  the  water  and  that 
information  is  relayed  in  a  visual  format  to  the  radar  operator.  When  used  in  conjunction 
with  nautical  charts,  these  radar  readings  can  help  determine  the  precise  location  of  the 

ship.  This  capability  is  critical  in  cases  where  GPS  satellite  signals  are  lost. 
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(2)  Automatic  Identification  System 

Automatic  Identification  System  (AIS)  is  a  system  that  uses  GPS  to  traek  and 
display  oeean  vessels.  The  system  is  an  international  standard  that  is  used  by  nearly  all 
large  commercial  shipping  traffic.  U.S.  Title  33,  Code  of  Federal  Regulations  requires  all 
tankers,  all  vessels  certified  to  earry  over  150  passengers,  and  all  ships  over  300  gross 
tons  to  have  the  system  installed  and  aetivated  (U.S.  Coast  Guard  Navigation  Center, 
2015).  This  policy  is  enforced  by  the  U.S.  Coast  Guard.  The  data  collected  by  the  system 
is  overlaid  onto  an  electronic  chart  display  and  information  system.  The  system  displays 
vessels  as  symbols  which  can  be  selected  to  show  more  information  about  any  speeific 
ship.  AIS  keeps  a  reeord  of  the  vessel  name,  eall  sign,  voyage  plan,  maneuvering 
information,  and  other  data  that  ean  be  used  by  mariners.  Oftentimes,  AIS  is  used  by  the 
U.S.  Navy  to  find  surfaee  contaets  before  radar  ean  reaeh  them.  The  inclusion  of  call 
signs  in  the  system  lets  mariners  contact  each  other  over  the  radio  and  arrange  safe 
passage. 

(3)  Eleetronie  Chart  Display  and  Information  System 

Eleetronie  Chart  Display  and  Information  System — ^Navy  (ECDIS-N)  is  the  U.S. 
Navy’s  primary  shipboard  navigation  tool  (Chief  of  Naval  Operations,  2001).  Overlaying 
the  ship’s  own  internal  sensors  and  AIS  data  onto  National  Geospatial  Agency  released 
digital  nautical  charts,  navigation  officers  are  provided  a  real-time  pieture  of  what  is 
proximate  to  the  ship.  This  same  system  enables  the  navigation  team  to  plan  and  execute 
transits  with  great  preeision.  By  plaeing  waypoints  into  the  system,  a  safe  transit  that 
avoids  land  and  shoal  water  is  created.  When  the  ship  executes  these  transits,  the  system 
provides  constant  guidance  of  how  to  maintain  and  regain  the  planned  traek. 

A  key  feature  of  the  system  is  the  digital  chart  correction  component.  Whereas 
with  paper  charts,  ship  personnel  were  required  to  read  lengthy  reports  and  manually 
draw  in  correetions  on  the  charts,  the  ECDIS-N  system  allows  for  instant  importation  of 
chart  updates.  This  not  only  ensures  eonsistency  in  the  chart  updates,  but  also  saves  many 
man-hours  related  to  chart  corrections. 
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C.  VIRTUAL  TESTBED  TECHNOLOGY 

1.  Virtual  Reality 

Professor  Frederick  P.  Brooks,  Jr.  defines  a  virtual  reality  (VR)  experience  as 
“any  in  which  the  user  is  effectively  immersed  in  a  responsive  virtual  world”  (1999,  p. 
16).  Additionally,  Bryson  (1996,  p.  62)  specifies  the  need  to  create  worlds  “in  which  the 
user  interacts  directly  with  virtual  objects.”  Any  definition  given  by  subject  matter 
experts  will  cover  a  broad  category  of  applications;  however,  the  general  consensus  is 
that  VR  has  an  experiential,  rather  than  technological,  focus  designed  to  draw  the  user  out 
of  the  real  world  and  place  them  into  a  virtual  one  (Steuer,  1992). 

While  modeling  and  simulating  tasking  can  never  fully  replace  the  true  physical 
undertaking,  there  is  ample  evidence  of  the  power  and  capability  that  VR  technology 
brings  to  the  realms  of  training  and  operational  testing.  Today’s  VR  systems  are  being 
used  in  the  fields  of  education,  entertainment,  industry,  architecture,  cultural  heritage, 
medicine,  psychology,  and  even  the  military.  As  an  example,  VR  studies  and  applications 
have  been  used  to  treat  phantom  pains  in  amputee  victims,  sooth  burn  victim’s  pain 
during  recovery  therapy,  assist  disabled  patients  with  physical  therapy,  and  even  provide 
treatment  to  soldiers  suffering  from  post-traumatic  stress  disorder  (PTSD)  (Casti,  2014; 
Rizzo,  Difede,  Rothbaum,  Daughtry,  &  Reger,  2013;  Yano,  Tamefusa,  Tanaka,  Saito,  & 
Iwata,  2012;  Rothbaum  et  ah,  1999).  Additionally,  VR  has  allowed  surgical  students  to 
practice  their  operating  room  techniques  and  has  even  given  autistic  children  a  medium  in 
which  they  can  better  learn  social  and  motor  skills  (Casti,  2014;  Maskey,  Lowry, 
Rodgers,  McConachie,  &  Parr,  2014;  Seymour  et  ah,  2002). 

Interactive  3D  video  games  are  the  best  examples  of  VEs  that  allow  multiple 
users  to  control  characters  and  character-based  interactions  within  the  virtual  space. 
Beyond  simple  entertainment,  multi-user  virtual  environments  (MUVE)  have  been  shown 
to  support  teaching  and  learning.  As  an  example.  Harvard  University’s  “River  City  is  a 
MUVE  for  teaching  scientific  inquiry  and  21st-century  skills  in  middle  school  science 
classes”  (Dieterle  &  Clarke,  2007,  p.  2).  This  system  has  been  proven  to  allow  for  the 
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monitoring  and  assessing  of  individual  students  far  better  than  the  traditional  teaching 
systems. 

2,  Augmented  Reality 

Whereas  VR  attempts  to  fully  encompass  users  in  a  VE,  AR  attempts  to  display 
only  elements  of  the  VE  to  the  users.  To  describe  AR  is  to  describe  a  technology  that 
“supplements  the  real  world  with  virtual  (computer-generated)  objects  that  appear  to 
coexist  in  the  same  space  as  the  real  world”  (Krevelen  &  Poelman,  2010,  p.  1).  Just  as  in 
VR,  this  is  accomplished  by  the  use  of  special  display  solutions  such  as  headsets,  display 
monitors,  and  programmable  glass.  The  critical  requirements  for  an  AR  system  are: 
combining  real-world  objects  with  virtual  overlays,  properly  aligning  those  objects,  and 
being  interactive  in  real-time  (Krevelen  &  Poelman,  2010).  Additionally,  true  AR 
systems  must  be  registered  in  three  dimensions  (Azuma,  1997). 

With  exception  of  the  overlay  requirement,  these  rules  apply  to  VR  just  the  same. 
The  difficulty  in  AR  however  comes  from  alignment,  i.e.,  registration  of  components  of 
virtual  and  real  world.  While  VEs  are  fully  defined  in  their  shape,  size,  colors,  textures 
and  behaviors;  the  ability  to  align  virtual  objects  precisely  over  the  (unknown)  real  world 
is  a  non-trivial  matter.  A  relaxed  definition  of  AR  removes  the  difficult  factor  of 
alignment.  Instead,  the  augmented  layer  can  simply  overlay  the  user’s  visual  field  with 
digital  information,  ignoring  the  need  to  track  any  real-world  objects. 

As  display  technologies  and  computers  become  faster  and  cheaper,  AR  will 
become  more  useful.  The  concept  of  utilizing  AR  in  the  maritime  domain  is  not 
unprecedented.  There  already  exists  patents  for  such  systems;  more  specifically  there  is 
one  for  a  “Wearable  marine  heads-up  display  system”  that  outlines  many  of  the  ideas 
presented  in  the  Chapter  I  (U.S.  Patent  No.  20070030211  Al,  2006,  p.  1).  Military  use  of 
AR  is  also  not  unique  as  the  newest  fighter  aircraft,  the  P35  Eighting,  features  an 
AN/AAQ-37  Distributed  Aperture  System  (DAS).  This  video-see-through  headset  system 
allows  the  pilot  to  not  only  look  down  through  the  physical  cockpit  onto  the  ground,  but 
also  displays  a  clear  bright  landscape  at  night  and  can  simultaneously  overlay  target 
acquisition  data  (Northrop  Grumman,  n.d.).  With  military  AR  systems  already  being 
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utilized  in  real-world  operations,  aeoeptanee  of  sueh  teehnologies  in  other  domains,  sueh 
as  shipboard  navigation,  may  follow  the  same  adoption  path. 

3,  Head-Mounted  Displays 

VR  teehnology  has  been  around  for  more  than  four  decades  and  during  that  time 
VR  systems  have  used  a  rich  variety  of  visual  display  solutions.  The  recent  boom  in  cell 
phone  technology  has  allowed  engineers  to  begin  bringing  VR  to  the  consumer  market 
through  cheaper  and  more  effective  head-mounted  displays  (HMD).  These  HMDs  are 
“interactive  head-referenced  computer  displays  that  give  users  the  illusion  of 
displacement  to  another  location”  (Ellis,  1994,  p.  17).  Depending  on  the  level  of  haptic 
sensory  information  and  modality  of  user  interaction  within  the  system,  the  environment 
can  allow  user  interaction  with  the  virtual  objects  comparable  to  interactions  one  would 
have  in  the  real  environment.  The  trends  of  increased  wireless  availability, 
miniaturization  of  electronic  sensors,  and  higher  resolution  display  technology  are 
collectively  allowing  HMDs  to  be  used  in  our  daily  lives.  (Cakmakci  &  Rolland,  2006) 

As  a  system,  HMDs  comprise  of  several  components.  The  following  is  a 
description  of  each  of  the  key  mechanisms  that  make  up  an  HMD. 

(1)  Display 

The  primary  visual  component  of  a  HMD  is  the  display.  Older  hardware  used 
miniature  monochrome  cathode  ray  tube  (CRT)  displays  but  they  tended  to  be  heavy, 
large,  and  required  significant  power  to  run  (Patterson,  Winterbottom,  &  Pierce,  2006). 
Today’s  displays  utilize  liquid-crystal  display  (LCD)  and  light-emitting  diode  (LED) 
panels  making  them  lighter,  smaller,  and  less  power  intensive. 

An  important  distinction  between  HMD  displays  is  the  field  of  view  (LOV)  that 
they  offer.  The  average  human  has  a  natural  LOV  of  “200°  horizontal  by  130°  vertical, 
with  the  central  120°  being  the  area  of  binocular  overlap”  (Patterson  et  ah,  2006,  p.  562; 
Velger,  1998,  p.  50;  U.S.  Department  of  Defense,  1962).  This  large  LOV  is  not  necessary 
to  replicate  for  all  applications  but  for  realistic  environment  trainers,  the  display  should 
provide  as  close  to  human  peripheral  vision  as  possible  (Cakmakci  &  Rolland,  2006). 
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The  commercial  displays  available  today  offer  FOV  that  range  from  100°  to  164° 
horizontal  and  up  to  60°  vertical  (Sensics,  n.d.;  Oculus  VR,  n.d.). 

Other  critical  components  of  the  display  are  luminance  and  resolution.  Both  of 
these  elements  provide  the  HMDs  with  the  capability  to  render  images  that  are  clearly 
visible  with  high  contrast  (Patterson  et  ah,  2006).  Luminance  is  determined  by  the  display 
material  (CRT,  LCD,  LED)  and  can  range  from  5,000  fL  to  12,000  fL  for  high-end 
displays  (Velger,  1998).  Resolution  is  immediately  perceived  in  the  display  due  to  the 
proximity  of  the  eye  to  the  screen  when  wearing  an  HMD.  Today’s  LED  screens  allow 
for  up  to  1920x1200  pixels  per  eye  (Sensics,  n.d.). 

(2)  Head  Tracking 

Tracking  the  rotation  of  the  user’s  head  is  a  significant  characteristic  that  separate 
HMDs  from  simple  monitors  or  televisions.  When  a  human  moves  and  rotates  their  head 
in  any  direction,  the  eyes  are  presented  with  a  new  view.  To  replicate  this  effect,  VR 
systems  track  head  position  and  rotation  about  the  X,  Y,  and  Z  planes.  General  head 
tracking  can  also  include  positional  tracking  which  provides  a  full  six  degree  of  freedom 
(DOE)  for  the  user.  The  rendered  scene  will  shift  in  correlation  with  the  head  movement. 
The  level  of  tracking  precision  for  a  system  determines  if  minute  shifts  in  the  head’s 
position  are  represented  in  the  scene.  In  addition  to  precision,  the  latency  of  tracking 
updates  is  a  considerable  issue  that  affects  every  HMD  design.  Delays  in  rendering  the 
scene  with  respect  to  the  head  tracking  data  can  cause  significant  health  issues  including 
cybersickness,  which  is  covered  later  on  in  this  chapter  (Patterson  et  ah,  2006). 

(3)  Erame 

The  frame  of  the  HMD  is  an  ergonomical  consideration,  particularly  with  regards 
to  its  weight.  As  a  device  for  research,  training,  or  entertainment,  HMDs  are  being 
designed  for  extended  use.  The  effects  of  cybersickness  aside,  discomfort  from  the 
weight  of  the  headset  can  be  a  limiting  factor  to  the  user  experience.  Prolonged  exposure 
to  excessive  weight  on  the  head  can  cause  shoulder  and  neck  irritation.  Additionally, 
because  fully  enclosed  HMDs  completely  cover  the  face,  heat  can  build  up  and  cause 
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sweating  or  fogging  of  the  lenses.  Excessive  sweat  in  the  mask  is  a  sanitary  issue  that 
needs  to  be  addressed  if  conducting  subject  testing  with  HMDs. 

4,  Testing  and  Measuring  Performance 

While  AR  and  VR  do  offer  a  wide  range  of  possibilities  to  several  domains,  there 
still  exist  several  critical  areas  of  study  within  these  technologies  that  have  yet  to  be  fully 
understood.  These  include  testing  and  measuring  human  performance,  measuring  sense  of 
presence,  measuring  navigational  complexity  of  the  environment,  and  estimating 
distances  in  VEs.  Each  of  these  issues  has  been  researched  in  depth,  but  just  as  every  AR 
or  VR  system  has  a  particular  combination  of  different  technical  characteristics  and 
capabilities  that  are  unique,  so  too  are  the  effects  they  may  have  on  human  operators. 
What  is  important  to  take  away  is  not  a  “one-size-fits-all  solution”  on  how  to 
accommodate  for  these  issues,  but  rather  a  general  understanding  of  what  to  look  for 
when  designing  such  systems.  With  that  in  mind,  the  first  topic  to  be  discussed  will  be  the 
testing  and  measuring  of  human  performance. 

(1)  Benchmark  Performance 

One  of  the  most  difficult  aspects  of  testing  and  evaluating  the  effectiveness  of  any 
virtual  system  is  how  to  define  good  performance  measurements  and  accurately  collect 
them.  Before  any  testing  or  performance  measurements  begin,  one  has  to  be  reminded  of 
the  results  supported  by  numerous  studies  in  the  domain  of  learning  and  training,  “Some 
tasks  may  be  uniquely  suited  to  virtual  representation  while  others  may  not  be  effectively 
performed  in  such  environments.”  (Stanney,  Mourant,  &  Kennedy,  1998,  p.  330).  To 
even  begin  measuring  the  effectiveness  of  a  VE,  one  must  ascertain  a  means  to  assess  the 
human  performance  in  the  virtual  world.  (Stanney,  et  ah,  1998)  Given  that  requirement, 
there  are  three  major  factors  that  are  found  to  influence  human  performance  in  VR 
(shown  in  Eigure  1):  (1)  Navigational  complexity,  (2)  presence,  and  (3)  benchmark 
performance  (Stanney  et  ah,  1998).  Benchmark  performance  is  a  unique  measurement 
determined  by  the  application  and  the  user’s  baseline  capabilities.  More  interesting  are 
the  variables  of  navigational  complexity  and  presence  which  can  be  manipulated. 
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Figure  1 .  “Components  of  Human  Performance  in  Virtual  Environments” 

(from  Stanney  et  al,  1998,  p.  329) 


(2)  Measuring  Presence 

“Presence  is  defined  as  the  subjective  experience  of  being  in  one  place  or 
environment,  even  when  one  is  physically  situated  in  another”  (Witmer  &  Singer,  1998, 
p.  225).  While  some  proponents  focus  on  the  visual  stimuli  of  these  environments,  many 
researchers  counter  that  presence  is  more  about  the  user  being  able  to  do  things;  arguing 
that  “the  VE  becomes  endowed  with  ‘there-ness’  through  this  process  of  action  and 
interaction”  (Slater  &  Steed,  2000).  The  consensus  among  researchers  in  this  domain 
suggests  that  presence  is  not  an  empirical  factor  that  can  be  easily  measured.  The  same 
researchers  propose  different  means  to  measure  this  phenomenon.  For  example,  reflexive 
responses,  a  three- attribute  subjective  category  rating  scale,  and  measurement  based  on 
discrimination  are  all  methods  suggested  by  Sheridan  (1994).  Slater  and  Steed  suggest 
adding  understanding  “based  on  data  that  can  be  unobtrusively  obtained  during  the  course 
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of  a  VE  experience”  (2000,  p.  413).  Slater  extends  this  further  by  codifying  another 
contributing  factor  to  presence.  This  factor  is  one  he  calls  “Plausibility  Illusion” 
referencing  that  users  in  a  VE  actually  believe  that  the  scenario  they  are  experiencing  is 
really  happening  (2009).  Additional  measures  have  been  obtained  using  different  sensors 
that  determine  physiological  responses  of  a  human  operator  (Meehan,  Razzaque, 
Whitton,  &  Brooks  Jr,  2003).  In  their  work  on  measuring  the  effects  of  latency  on 
presence  in  stressful  VEs,  they  measured  heart  rate  and  skin  conductance.  Given  the 
numerous  definitions  or  measurements  being  used  to  study  the  field  of  VR,  it  is 
surprising  that  such  things  have  not  been  standardized.  As  such  it  will  continue  to  be  the 
individual  responsibility  of  researchers  to  maintain  awareness  of  current  practices. 

(3)  Measuring  Navigational  Complexity 

In  measuring  navigational  complexity  in  their  seminal  study  on  navigating  VEs 
Usoh  et  al.  used  the  factors  of  “How  Simple?”;  “How  Straightforward?”;  and  “How 
Natural?”  to  score  the  ease  of  locomotion  (1999).  Navigation  through  the  environment  is 
determined  by  the  mechanism  of  modality.  Whether  it  is  full  motion  tracking  via  sensors, 
a  multidirectional  treadmill,  or  even  a  game  controller;  users  must  be  trained  on  the 
proper  method  to  move  about  the  scene.  As  walking  is  a  natural  task  for  humans,  it  is  one 
of  the  least  complex  navigational  methods.  Tracking  users  as  they  walk  through  the 
physical  environment  has  been  proven  to  significantly  increase  their  level  of  presence  in 
the  VE  (Usoh  et  ah,  1999).  Unless  highly  familiar  with  the  system,  when  users  are 
required  to  use  devices  such  as  game  controllers  to  maneuver  their  position  in  the  VE 
their  cognitive  processing  is  split.  This  split  disrupts  the  user’s  ability  to  be  immersed  in 
the  VE,  thus  lowering  their  overall  performance. 

(4)  Estimating  Distance  in  Virtual  Environments 

Studied  extensively,  the  cognitive  issue  of  estimating  distances  in  VEs  is  currently 
a  serious  limitation  of  VR,  especially  in  situations  when  this  will  influence  basic  task 
performance.  Whereas  humans  can  effectively  estimate  distances  out  to  20  meters,  doing 
so  when  immersed  in  a  VE  is  challenging  and  thus  less  accurate  (Thompson  et  ah,  2004). 
Depending  on  the  purpose  of  the  simulation,  being  able  to  develop  spatial  cognition  of 
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the  scene  may  be  necessary.  Researchers  agree  that  accurately  estimating  distance  is 
necessary  to  form  an  accurate  mental  model  of  the  environment  (Popp,  Platzer,  Eichner, 
&  Schade,  2004;  Witmer  &  Kiline,  1998).  It  is  for  this  reason  that  studies  have  been 
conducted  to  understand  why  estimation  of  distance  in  VR  is  difficult. 

Judging  the  distance  of  an  object  in  VR  usually  requires  that  the  subject  be 
familiar  with  the  relative  size  of  a  similarly  shaped  object.  Studies  have  shown,  however, 
that  “changes  in  object  size  may  be  misinterpreted  as  changes  in  distance  and  vice  versa” 
(Witmer  &  Kiline,  1998,  p.  146).  The  causation  for  this  issue  is  not  isolated  to  just  one 
factor  and  thus  there  is  no  agreement  from  researchers  as  to  the  underlying  fundamental 
reason  why  distance  estimation  is  so  difficult  in  VR.  Poor  resolution,  low  FOV  angles, 
inadequate  tracking,  and  binocular  depth  cues  are  all  contributing  factors  to  this  problem. 

One  proposed  method  for  increasing  the  ability  to  estimate  distances  in  VR  is  by 
rendering  a  “fully-articulated  and  tracked  visual  representation”  of  the  user  (Mohler, 
Creem-Regehr,  Thompson,  &  Bulthoff,  2010,  p.  230).  By  increasing  the  correlation  with 
an  avatar  in  the  VE,  users  were  able  to  more  successfully  judge  the  environment  around 
them.  This  is  an  important  discovery  which  compliments  the  findings  by  Thompson  et  al. 
that  increasing  the  graphic  fidelity  alone  does  not  necessarily  increase  distance  estimation 
capabilities  (2004).  The  final  solution  for  giving  humans  the  capability  to  accurately 
estimate  distance  in  VR  will  likely  come  from  integrating  a  combination  of  multiple 
techniques. 

5.  Health  and  Safety  Considerations  in  Virtual  Reality  Systems 

(1)  Cybersickness 

Thus,  far,  the  discussion  has  been  relegated  to  the  technical  characteristics  and 
performance  characteristics  of  VR  and  AR.  Another  critical  aspect  of  these  fields  is  the 
health  and  safety  issues  associated  with  research  in  these  domains.  These  include  things 
such  as  cybersickness,  simulator  sickness,  cognitive  tunneling,  and  other  physical  safety 
concerns.  The  rest  of  this  chapter  will  focus  primarily  on  the  effects  and  causations  of 
cybersickness  as  well  as  other  safety  concerns. 
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Commonly  associated  or  confused  as  motion  sickness,  cybersickness  is  a  direct 
byproduct  of  VR  development.  While  sharing  symptoms  of  nausea,  drowsiness, 
headaches,  disorientation,  fatigue,  and  many  more,  the  cause  of  cybersickness  is  different 
than  that  of  simulator  sickness  (Stanney,  Kennedy,  &  Drexler,  1997).  Whereas  vestibular 
stimulation  by  itself  can  cause  simulator  sickness,  cybersickness  can  be  brought  on  solely 
with  visual  stimulation.  (LaViola,  2000).  Given  these  differences,  a  clear  line  has  been 
drawn  between  the  two  but  the  underlying  causations  of  each  are  very  similar.  The 
standard  theory  of  causation  for  motion  sickness  from  VR  simulations  is  suggested  in  the 
sensory-conflict  theory  (Cobb,  Nichols,  &  Ramsey,  1999).  “Sensory-conflict  theory 
proposes  that  symptoms  occur  as  a  result  of  conflict  between  signals  received  by  the  three 
major  spatial  senses:  the  visual  system,  the  vestibular  system,  and  nonvestibular 
proprioception”  (Cobb  et  ah,  1999,  p.  170).  To  simplify,  the  human  body  gets  sick 
because  the  brain  cannot  process  the  differences  between  what  it  is  seeing  and  feeling. 
This  leads  to  vestibular  instability,  visual  fluxes,  and  other  physical  manifestations.  No 
matter  the  cause,  the  symptoms  outlined  above  tend  to  last  less  than  a  couple  of  hours  and 
have  not  been  proven  to  cause  long-term  harm. 

There  are  several  specific  influences  that  have  been  proven  to  raise  the  chances  of 
cybersickness.  For  instance,  female  subjects  are  more  at  risk  than  men  due  to  their  wider 
FOV  which  increases  flicker  perception.  (LaViola,  2000)  Another  significant  factor  that 
is  relevant  to  any  future  study  in  shipboard  navigation  would  be  speed.  Testing  has 
shown  that  “vection  sensation  and  sickness  symptoms  increased  with  increasing 
navigation  speeds  from  3m/s  to  lOm/s”  within  the  simulation  (So,  Lo,  &  Ho,  2001). 
Vection  in  this  situation  is  defined  as  “visually  induced  illusory  self-motion”  (Hettinger, 
Berbaum,  Kennedy,  Dunlap,  &  Nolan,  1990,  p.  171).  Being  one  of  the  direct  causations 
of  sensory-conflict  it  must  be  taken  into  account  in  motion  platform  systems.  As  any 
simulation  allowing  for  shipboard  navigation  will  include  speed  changes  ranging  from  0 
to  15  m/s,  vection  must  be  a  consideration  at  the  design  stage. 

(2)  Cognitive  Tunneling 

Commonly  associated  with  AR  applications,  cognitive  tunneling  can  be  a  serious 

health  and  safety  threat  to  AR  users.  Cognitive  tunneling  is  defined  as  the  effect  in  which 
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a  user  focuses  attention  on  a  specific  visual  sector  and  therefore  loses  peripheral  focus  on 
other  visual  elements  (Thomas  &  Wickens,  2001;  Dowell,  Foyle,  Hooey,  &  Williams, 
2002;  Crawford  &  Neal,  2006).  The  result  is  a  distracted  user  who  loses  focus  of  the 
overall  task.  In  aviation,  cognitive  tunneling  has  been  associated  with  pilots  who  focus  on 
data  readings  in  their  HUD  rather  than  the  environment  around  them  (Dowell  et  ah, 
2002).  In  automobiles,  the  effect  can  be  seen  by  drivers  paying  attention  to  CD  players, 
GPS,  and  other  systems  rather  than  the  road  in  front  of  them  (Olsson  &  Burns,  2000). 
Although  not  guaranteed  to  occur,  cognitive  tunneling  must  be  accounted  for  when 
integrating  AR  into  existing  systems. 

(3)  Other  Safety  Issues 

Beyond  the  issue  of  cybersickness,  there  are  other  safety  considerations  that  must 
take  place  when  working  with  VR,  and  especially  with  an  HMD.  The  first  lies  with  the 
physical  hardware.  Heavy  and  bulky  headsets  with  prolonged  usage  are  known  to  cause 
discomfort  to  subjects  and  can  cause  adverse  effects  on  the  experiment’s  results  (Cobb, 
Nichols,  &  Ramsey,  1999).  Cable  management  can  also  be  a  significant  factor.  By  using 
headsets  with  tethered  cables,  subjects  can  become  entangled  and  trip,  causing  not  only 
harm  to  the  subject,  but  also  possibly  negating  results  in  the  simulation  or  damaging 
equipment.  On  that  same  note,  space  allocation  is  another  critical  component  to  running 
experiments  in  VR.  “Obviously,  HMD  use  should  not  take  place  in  areas  where  there  are 
any  hazardous  objects  or  substances  as  the  participant  is  unable  to  see  their  real-world 
surroundings  and  may  be  at  risk  of  injury”  (Cobb  et  ah,  1999,  p.  183).  Overall,  it  is  a 
combination  of  physical  and  physiological  health  issues  that  must  be  taken  into 
consideration  when  working  with  VR  or  AR. 

D.  SUMMARY 

This  chapter  has  covered  multiple  domains  ranging  from  military  operations  to 
academic  research  in  the  fields  of  VR  and  AR.  With  all  of  these  topics  being  researched 
well  ahead  of  the  design  stage  of  the  study,  many  of  the  pitfalls  associated  with  VR  and 
AR  applications  were  taken  into  consideration.  The  analysis  of  the  literature  helped  to 
craft  a  better  experience  for  the  test  subjects;  some  of  these  decisions  included  the  way  a 
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HUD  layout  was  manipulated  to  reduce  cognitive  tunneling  and  the  adjustment  of  the 
ship’s  speed  to  prevent  cybersickness. 
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III.  DESCRIPTION  OF  SHIP  NAVIGATION  RESPONSIBILITIES 


This  chapter  details  the  roles,  responsibilities,  and  tasks  performed  by  the  primary 
individuals  associated  with  the  operation  of  safe  navigation  of  a  U.S.  Navy  warship. 
While  there  are  many  other  eritieal  members  of  the  navigation  team  on  the  bridge,  this 
ehapter  foeuses  on  the  roles  of  helmsman,  navigation  evaluator,  and  conning  officer. 
Eaeh  of  these  individuals  has  his  own  unique  duties  to  perform.  When  their  tasks  are  well 
integrated  and  well  executed,  the  combined  performanee  results  in  the  safe  navigation  of 
the  ship.  While  no  two  ships  within  the  U.S.  Navy  operate  in  exaetly  the  same  manner, 
these  three  roles  are  largely  standardized  throughout  the  fleet  to  deliver  safe  navigation. 

Driving  warships  in  today’s  complex  taetieal  environments  requires  highly  skilled 
personnel.  Whether  avoiding  unmarked  fishing  buoys,  maneuvering  for  air  operations,  or 
maintaining  taetieal  formations;  the  basie  tenants  of  driving  warships  apply  to  every  size 
of  vessel  in  the  Navy’s  collection.  While  there  are  a  number  of  situations  where  the 
members  of  the  bridge  team  must  complete  speeific  event-dependent  tasking,  the  study 
we  condueted  focused  on  three  members  of  the  bridge  team  during  a  routine  ehannel 
transit.  The  reason  the  study  foeused  on  this  situation  is  beeause  restrieted  water 
navigation  relies  heavily  on  the  skills  of  the  eonning  officer  and  the  doetrine  for  these 
operations  is  thoroughly  doeumented.  Therefore,  the  following  seetions  outline  the 
general  tasks  and  expectations  of  each  of  these  members  during  such  a  transit. 

A.  RESPONSIBILITIES  OF  A  HELMSMAN 

An  important  member  of  the  bridge  team,  the  inelusion  of  the  helmsman  during 
testing  amplified  the  realism  of  the  study  and  enabled  the  subjeets  to  drive  the  ship  as 
they  would  in  real  life.  Whereas  the  tasks  required  of  the  navigator  and  conning  officer 
have  ehanged  slightly  with  teehnology  advancements,  the  job  of  the  helmsman  has 
remained  constant:  he  “steers  the  ship  as  directed  by  the  conning  officer”  (Cutler,  1998). 
The  standardized  method  of  completing  that  tasking  is  a  multi-step  process. 

1 .  Repeat  baek  order  as  given  (verbatim) 

2.  Shift  rudder  aceording  to  order 
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3.  Report  the  status  of  the  rudder  once  it  is  in  position 

4.  Manipulate  the  rudder  so  that  the  ship’s  heading  matches  the  ordered 
course 

5.  Report  that  the  ordered  course  has  been  met 

6.  Maintain  the  ordered  course 

There  are  many  special  circumstances  in  which  the  steps  differentiate;  however, 
the  general  list  of  steps  that  we  provided  accounts  for  most  situations.  Whereas  there  is 
always  a  helmsman  at  the  helm  while  the  ship  is  at  sea,  it  is  common  practice  to  put  the 
best  and  most  experienced  helmsman  on  the  bridge  team  during  a  restricted  maneuvering 
situation.  This  is  because  restricted  maneuvering  requires  precision  ship  driving,  loud  and 
clear  repeat-backs,  and  the  capability  to  instantly  take  corrective  actions  in  cases  of 
emergency. 

A  helmsman’s  skill  is  not  only  determined  by  quick  translation  of  commands  to 
actions,  but  also  by  the  ability  to  maintain  a  steady  course.  The  addition  of  winds  and 
currents  are  what  differentiate  driving  a  ship  from  driving  a  car.  Whereas  a  well-aligned 
car  can  maintain  its  direction  on  a  straight  road,  the  helmsman  must  constantly  make 
minor  rudder  adjustments  to  maintain  course.  In  the  case  where  the  ship  is  being  pushed 
significantly,  more  action  is  required  by  the  helmsman. 

Generally,  there  are  limits  placed  on  how  much  rudder  the  Helmsman  can 
use  without  first  getting  permission  from  the  Conning  Officer.  For 
example,  in  most  cases,  no  more  than  ten  degrees  rudder  in  either  direction 
is  allowed  without  first  requesting  permission  from  the  Conning  Officer. 

The  primary  reason  the  Helmsman  would  need  to  use  more  than  ten 
degrees  rudder  to  maintain  a  course  is  a  high  sea-state.  In  this  case,  the 
Helmsman  requests  permission  to  use  more  than  ten  degrees  rudder  to 
maintain  course.  The  Conning  Officer  would  likely  grant  permission,  and 
then  the  Helmsman  would  be  free  to  use  the  rudder  as  required.  (Norris, 

1998,  p.  96) 

Although  most  of  the  actions  of  a  helmsman  are  scripted,  it  is  also  the 
helmsman’s  job  to  ensure  (check)  that  orders  given  to  him  do  make  sense.  An  example 
would  be  for  the  conning  officer  to  give  a  left  rudder  command  for  a  course  that  is  nearer 
to  starboard.  While  this  is  indeed  a  legitimate  order  that  can  be  executed,  it  is  more  likely 
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that  the  conning  officer  simply  made  a  mistake.  It  is  during  these  points  that  the 
helmsman  would  attempt  to  seek  clarification  by  asking  “Orders  to  the  helm?”  During  a 
channel  transit  with  many  quick  successive  turns  it  is  important  that  the  helmsman  be 
able  to  not  only  focus  on  the  precise  handling  of  the  ship  but  to  also  smartly  interpret  the 
commands  being  given  to  him. 

While  ship  driving  practices  are  changing  in  the  U.S.  Navy’s  newest  generation  of 
ship,  the  nature  of  how  the  bulk  of  the  U.S.  fleet  operates  at  sea  demands  that  the  position 
of  helmsman  be  filled  for  many  years  to  come  (Ewing,  2008).  It  is  a  combination  of  their 
quick  reaction  capabilities  and  a  practiced  skillset  of  handling  casualties  that  makes  the 
helmsman  such  a  vital  part  of  the  bridge  team.  It  is  for  this  reason  that  including  the 
helmsman  in  the  study  was  required. 

B,  RESPONSIBILITIES  OF  A  NAVIGATION  EVALUATOR 

A  critical  portion  of  the  study  replicated  the  everyday  interactions  between 
conning  officer  and  navigation  evaluator.  In  order  to  make  a  baseline  determination  of 
how  well  conning  officers  perform  in  the  current  operational  environment,  it  was 
essential  to  include  a  navigation  evaluator  as  well.  Usually  performed  by  the  ship’s 
navigation  officer,  the  position  of  navigation  evaluator  is  an  important  role  that  is 
responsible  to  the  commanding  officer  for  the  safe  navigation  of  the  vessel.  This 
responsibility  is  so  great  that,  in  the  event  of  extreme  danger,  “the  navigator  may  be 
authorized  by  the  captain  to  relieve  the  officer  of  the  deck  (OOD)  if  in  his  judgment  the 
OOD  is  endangering  the  ship  from  a  navigation  standpoint”  (Stavridis  &  Girrier,  2007). 
The  logic  behind  such  a  rule  is  that  the  navigator  is  oftentimes  a  highly  seasoned  officer 
with  more  training  and  experience  specifically  in  ship  driving  than  most  of  the  other 
watch  standers  on  the  bridge  team.  In  accordance  with  the  Navy’s  Navigation 
Department  Organizaion  and  Regulations  Manual  (NAVDORM),  the  navigation 
evaluator  is  primarily  responsible  for  using  all  gathered  electronic  and  visual  data  to 
determine  the  ship’s  position  and  make  recommendations  for  navigation 
(COMNAVSURFPAC,  COMNAVAIRPAC,  COMNAVAIRLANT,  & 
COMNAVSURFLANT,  2013). 
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Even  before  the  ship  begins  moving,  the  navigation  evaluator  has  a  medley  of 
duties  to  perform  on  the  bridge.  Once  a  destination  is  chosen,  be  it  a  port  or  open  sea,  the 
navigation  team  must  prepare  the  electronic  (or  paper)  charts.  This  process  involves 
updating  the  chart  for  recent  changes,  laying  down  tracks  that  are  both  safe  and  follow 
predetermined  shipping  lanes,  and  highlighting  predominant  landmarks  that  can  be  used 
as  visual  bearings.  The  NAVDORM  requires  that  a  navigation  brief  be  performed  no 
more  than  24  hours  before  entering  restricted  waters  (COMNAVSURFPAC  et  ah,  2013). 
This  brief  has  many  requirements,  but  the  primary  purpose  is  “to  provide  a  plan  for  safe 
and  prudent  passage,  including  piloting  in  restricted  waters”  (COMNAVSURFPAC  et  ah, 
2013).  Beyond  presentation  of  the  voyage  plan,  the  briefing  also  covers  the  expected 
weather,  traffic  conditions,  engineering  plant  status,  and  emergency  procedures  for  the 
operation.  Oftentimes,  the  navigator  will  work  with  the  conning  officer  well  before  the 
transit  to  ensure  he  understands  the  ship’s  entire  track. 

When  the  transit  begins,  the  navigation  evaluator  moves  to  the  plotting  table,  a 
desk  where  paper  or  electronic  charts  are  displayed,  and  begins  his  tasking.  At  this  point, 
he  must  ensure  fixes  are  being  taken  at  the  correct  intervals,  and  that  reports  are  being 
made  properly.  Fixes,  both  paper  and  electronic,  are  recorded  positions  of  where  the  ship 
is  at  a  specific  point  in  time.  This  can  be  calculated  by  GPS,  visual  bearings,  or  even 
radar.  Reporting  is  the  process  in  which  the  navigation  evaluator  vocally  indicates 
information  to  the  entire  bridge  team.  This  is  done  in  a  loud  and  clear  manner  so  that  each 
person  on  the  bridge  hears  the  report. 

Because  today’s  Navy  has  transitioned  primarily  to  electronic  navigation,  relying 
strictly  on  GPS,  the  requirements  of  the  navigation  evaluator  have  changed  drastically. 
Since  the  transition  is  not  complete  for  every  ship  yet,  the  old  methods  of  reporting  are 
still  be  used  by  a  few  ships  in  the  U.S.  Navy.  In  order  to  qualify  to  solely  use  electronic 
charts,  ships  must  receive  full  ECDIS-N  certification.  This  process  involves  equipment 
installation,  crew  training,  and  system  validation.  For  a  fully  qualified  vessel,  there  are  no 
structured  reporting  requirements  for  the  navigation  evaluator  with  the  exception  of 
emergencies  or  casualties.  Typically,  the  navigation  evaluator  will  let  the  conn  (conning 
officer)  know  if  the  ship  is  getting  too  far  off  the  track,  when  to  turn,  and  if  the  conn  has 
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ordered  a  eourse  that  is  heading  toward  dangerous  waters.  Additionally,  the  navigation 
evaluator  will  offer  eourse  reeommendation  to  regain  traek  or  eompensate  for  set  and 
drift.  The  reason  for  the  less  stringent  requirements  for  ECDIS-N  ships  is  because,  on 
most  vessels,  there  is  a  duplicate  navigation  display  system  in  the  middle  of  the  bridge. 
This  is  typically  where  the  conning  officer  will  stand,  and  therefore  he  can  look  down  at 
any  time  to  see  the  same  data  that  the  navigation  evaluator  is  looking  at. 

The  reporting  requirements  for  ECDIS-N  ships  are  fairly  open  and  situationally 
dependent;  however,  the  same  cannot  be  said  for  those  ships  not  qualified  for  ECDIS-N 
navigation.  Instead,  the  navigation  evaluators  on  these  ships  must  make  constant  reports 
at  regular  intervals  during  the  entire  restricted  water  transit.  Time  between  reports  is 
determined  by  the  ship’s  distance  from  shoal  water.  In  transiting  restricted  waters,  such 
as  the  scenario  used  in  the  study,  the  navigation  evaluator  would  make  reports  every  three 
minutes.  During  the  time  between  reports,  the  navigation  team  must  take  a  fix,  plot  it  on  a 
paper  chart,  compare  it  with  the  onboard  electronic  navigation  system,  and  make  the 
verbal  report.  This  report  is  highly  structured,  requiring  that  the  following  items  be 
included  (COMNAVSUREPAC  et  ah,  2013): 

•  Eix  Time 

•  Eix  Quality 

•  Eix  Method 

•  Eix  Position 

•  Recommendations 

•  Supplemental  Info  (  nearest  hazard/aid  to  navigation,  fathometer  reading, 
distance  and  time  till  turn,  next  course,  set  and  drift) 

After  every  report,  the  conning  officer  and  OOD  are  required  to  acknowledge 
their  receipt  and  understanding  of  the  information.  Additionally,  each  time  the  conning 
officer  orders  a  course  change,  the  navigation  evaluator  must  determine  if  the  course  is 
safe  for  navigation  and  report  this  information. 
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As  the  report  for  ships  not  eertified  to  rely  upon  ECDIS-N  is  made  at  a  regular 
interval  and  always  eontains  the  same  amount  of  information,  it  provides  a  better  testing 
parameter.  The  fact  that  there  still  remain  ships  that  are  not  qualified  for  ECDIS-N  made 
using  this  form  of  report  acceptable  for  our  testing  purposes. 

In  conclusion,  the  navigation  evaluator  is  a  key  member  of  the  bridge  team.  Even 
with  the  minimal  reporting  requirements  of  today’s  Navy,  conning  officers  still  rely 
heavily  on  the  reports  and  recommendation  of  the  navigation  evaluators.  Given  their 
years  of  shipboard  experience  and  specialized  training,  navigation  evaluators  are 
expected  to  be  able  to  deliver  safe  and  accurate  guidance  to  the  less  practiced  conning 
officers. 

C.  RESPONSIBILITIES  OF  A  CONNING  OFFICER 

Conning  is  the  act  of  controlling  the  physical  mechanisms  that  drive  the  ship 
(Stavridis  &  Girrier,  2007).  The  helmsman  is  an  intermediate  buffer  to  this  process,  but 
the  conning  officer  makes  the  tactical  level  decisions  that  enable  the  ship  to  navigate 
safely.  Primarily  through  multiple  opportunities  of  hands-on  training,  conning  officers 
develop  the  skill  of  taking  numerous  visual  and  aural  inputs  from  various  sources  and 
making  quick  decisions  that  keep  the  ship  safe.  Visual  inputs  include;  tracking  the 
relative  motion  of  the  ship  against  known  local  landmarks,  looking  at  the  water  for  signs 
of  active  currents,  and  monitoring  the  digital  navigation  equipment  on  the  bridge  such  as 
the  Voyage  Management  System  (VMS).  Aural  inputs  include;  listening  for  sound 
signals  from  approaching  vessels,  monitoring  bridge-to-bridge  radio  to  understand  what 
another  ship  is  doing,  and  paying  attention  to  the  navigator’s  recommendations.  These 
examples  only  encompass  a  fraction  of  the  total  number  of  elements  the  conning  officer 
needs  to  be  aware  of.  In  general,  conning  takes  a  significant  amount  of  focus  and  a 
goodly  amount  of  practice. 

Conning  is  a  very  situationally  dependent  task.  Depending  on  the  class  and 
mission  set  for  the  ship,  a  conning  officer  may  be  required  to  execute  an  underway 
replenishment  in  the  morning  and  then  drive  through  a  minefield  that  same  afternoon;  a 
multitude  of  operations  represent  a  reality  of  this  position.  The  goal  of  this  study  was  not 
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to  test  an  AR  display  for  every  possible  operation  a  conning  officer  can  be  involved  in. 
Instead,  this  study  focused  on  one  scenario  that  takes  place  every  time  a  ship  gets 
underway  from  port.  An  outbound  restricted  water  transit,  as  outlined  in  Chapter  II,  is  a 
common  task  that  every  experienced  conning  officer  must  be  familiar  with.  Therefore, 
the  rest  of  this  section  will  cover  the  basic  tasks  and  steps  that  a  conning  officer  would  be 
expected  to  perform  in  the  given  situation. 

Before  a  conning  officer  ever  assumes  the  watch,  he  is  required  to  fully 
understand  the  transit  plan.  The  plan  built  by  the  navigation  department  is  a  collection  of 
tracks  and  waypoints  overlaid  on  a  paper  or  digital  chart  that  dictates  how  the  ship  will 
proceed  out  to  sea.  Each  track  includes:  courses  and  distances  for  each  leg,  transit  speed, 
turning  points,  and  local  navigation  aids.  Having  a  clear  understanding  of  the  entire  plan 
allows  the  conning  officer  to  focus  on  dynamic  events  that  may  affect  the  transit.  These 
dynamic  events  include:  set  and  drift,  merchant  traffic,  adverse  weather  (i.e.,  fog),  and 
engineering  causalities.  After  the  conning  officer  has  familiarized  himself  with  the  transit 
plan,  the  ship  conducts  a  navigation  brief  that  details  all  watch  responsibilities  for  the 
operation.  At  this  point,  the  conning  officer  is  responsible  for  presenting  the  transit  plan 
to  all  parties  involved  in  the  evolution.  This  entire  formal  process  is  dictated  by  the 
NAVDORM,  and  is  required  before  any  restricted  water  transit  (COMNAVSURFPAC  et 
ah,  2013). 

The  conning  officer  is  responsible  for  issuing  commands  to  get  the  ship  away 
from  the  pier  and  into  the  channel  to  commence  the  transit.  While  issuing  commands  to 
the  helmsman,  the  conning  officer  uses  standard  verbiage  so  that  there  is  no 
misunderstanding  of  what  was  said  or  expected.  Once  away  from  the  pier  and  on  a  track, 
the  conning  officer  will  order  minor  course  corrections  to  compensate  for  set  and  drift  or 
to  avoid  minor  hazards  such  as  fishing  buoys.  As  the  next  course  approaches,  the  conning 
officer  will  check  their  bridge  wings  to  ensure  that  a  turn  will  not  cause  the  ship  to  run 
into  another  vessel  that  may  be  overtaking.  When  appropriate,  the  conning  officer  will 
order  the  helm  to  come  to  the  next  course  and  watch  the  rudder  indicator  for  evidence 
that  the  helmsman  is  properly  executing  the  order.  Watching  the  heading  change  on  the 
ship’s  gyrocompass  and  observing  the  surrounding  land  change  perspective  also  gives  the 
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conning  officer  an  indication  that  the  ship  is  turning  and  how  fast  it  is  doing  so.  This 
proeess  eontinues  until  the  ship  reaehes  the  end  of  the  navigator’s  traek  and  the  bridge 
team  begins  following  normal  at-sea  proeedures. 

A  eonning  offieer  demonstrates  his  skills  by  being  able  to  manipulate  the  ship’s 
engines  and  rudder  to  keep  the  ship  in  safe  water.  The  reason  why  the  term  “safe  water” 
is  used  in  lieu  of  driving  elose  to  traek  is  beeause  elosely  following  the  planned  traek  may 
not  be  possible  due  to  a  myriad  of  reasons,  sueh  as  other  vessel  traffic,  significant 
currents,  or  even  dynamie  seeurity  events.  Barring  unforeseen  oeeurrenees,  driving  close 
to  the  predefined  traek  provides  the  safest  method  of  transit.  In  order  to  stay  close  to  the 
traek,  a  eonning  officer  must  take  in  all  visual  and  audio  cues  available  and  output 
eorreetly  formatted  orders  that  either  regain  or  maintain  the  base  eourse.  This  is  done  by 
looking  at  how  far  off  the  traek  the  ship  is,  ealeulating  a  proper  eourse  to  regain  that 
traek,  and  then  compensating  for  set  and  drift.  Onee  all  the  ealeulations  are  eomplete,  the 
eonning  offieer  must  translate  what  they  want  to  do  into  the  proper  verbiage  to  give  elear 
orders  to  the  helmsman.  Being  able  to  quickly  and  accurately  eompute  these  ealeulations 
instantaneously  is  the  hallmark  of  a  skilled  conning  officer. 

Conning  offieers  must  also  be  able  to  eope  with  emergeney  situations  and  to  keep 
the  ship  in  safe  water  during  restricted  transits.  These  emergeney  situations  ean  inelude  a 
man  overboard,  engineering  easualties,  and  near  eollision  situations.  However,  sinee  the 
study  did  not  inelude  any  of  these  situations,  they  were  not  eovered  in  this  seetion. 

D.  SUMMARY 

This  ehapter  has  foeused  on  responsibilities  of  three  key  members  of  the  bridge 
team  during  a  routine  channel  transit.  There  are  several  other  operators  on  the  bridge 
during  a  restrieted  water  transit,  but  to  minimally  represent  the  ship  driving  aspects  of  the 
operation  these  three  positions  are  fundamental.  The  researeh  study  focused  on  measuring 
the  performanee  of  the  eonning  offieer  who  was  asked  to  perform  in  a  team  operation.  It 
was  therefore  important  to  properly  replieate  the  aetions  and  responses  of  the  other  two 
positions  so  that  test  subjeets  would  exeeute  their  tasks  most  effeetively  and  in  the  same 
manner  in  whieh  they  would  perform  on  a  real  ship. 
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IV.  EXPERIMENTAL  TESTBED  ENVIRONMENT 


This  chapter  details  the  teehnical  applications  and  processes  used  to  design, 
program,  and  simulate  the  testbed  environment.  Each  decision  made  during  the  design 
phase  had  a  significant  impact  on  how  final  testing  would  be  conducted.  Additionally,  the 
usefulness  and  aecuracy  of  the  colleeted  data  was  discussed  in  the  context  of  realism  and 
precision  of  the  simulation.  Overall,  the  ehapter  details  the  challenges  we  came  across, 
and  a  set  of  solutions  that  were  developed  and  tested  in  our  effort  to  ereate  the  most 
optimal  version  of  a  dynamic  real-time  ship  driving  simulation  that  would  support  user 
study. 

The  initial  discussions  and  considerations  focusing  on  the  prototyping  and  testing 
of  an  augmented  display  solution  for  conning  led  to  an  inevitable  conclusion.  Given  our 
need  for  repeatability,  safety,  and  the  ability  to  modify  the  software  and  use  it  in 
conditions  characteristic  for  AR,  it  would  be  a  nearly  impossible  task  to  conduct 
experimentation  onboard  an  actual  ship  busy  conducting  regular  operations.  Instead  of  a 
true  AR  setup,  a  decision  was  made  to  virtualize  the  entire  experimental  environment. 
This  deeision  coincided  with  a  simultaneous  emergence  of  a  cheap  VR  headset  in  the 
commereial  market  and  thus  led  to  the  decision  to  develop  and  operate  completely  in  VR 
using  a  head-mounted  display  solution. 

A,  HARDWARE 

1,  Immersive  Display  Solution 

We  deeided  to  use  the  Oeulus  Rift  HMD  as  our  immersive  display  for  this 
experiment.  Historically,  the  price  of  a  good  VR  headset  had  been  quite  prohibitive  for 
most  applications.  Additionally,  the  headsets  themselves  were  bulky,  heavy,  and  difficult 
to  program  for,  limiting  their  usefulness  as  a  researeh  platform.  In  eontrast,  the  Oeulus 
Rift  took  advantage  of  the  recent  development  of  fast  and  eheap  mobile  displays.  The 
eompany  declared  its  vision  to  be  a  distribution  of  “immersive  virtual  reality  teehnology 
that’s  wearable  and  affordable”  (“About  Oeulus,”  2015).  By  March  2014,  it  has 
commercially  released  two  development  kits,  eaeh  kit  eonsisting  of  a  head-mounted 
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display,  appropriate  cabling,  and  software  integration  packages.  The  first  development  kit 
(DKl)  was  released  in  late  2013  while  the  second  development  kit  (DK2)  was  released  in 
March  2014  (“Oculus  VR,”  n.d.).  A  comparison  of  the  technical  specifications  of  each  is 
shown  in  Table  1. 


PARAMETER: 

DKl 

DK2 

Resolution  (per  eye) 

600  X  800 

960  X 1080 

Display  Technology 

LCD 

OLED 

Display  Refresh  Rate 

60  Hz 

60,  72,  75Hz 

Persistence 

3ms 

2  ms,  3  ms,  full 

Visual  FOV 

110° 

100° 

Tracking  DOF 

3 

6 

Weight 

.83  lbs 

.97  lbs 

Table  1.  HMD  Comparison  (after  http;//riftinfo.com/oculus-rift-specs-dkl- 
vs-dk2-comparison,  http://in2gpu.eom/2014/08/10/oculus-rift-dkl-vs- 
dk2/,  and  https://www.oculus.com/dk2) 


(1)  DKl 

The  first  hardware  released  by  Oculus,  the  DKl,  was  a  low  resolution  headset 
with  more  focus  being  put  on  the  tracking  and  software  development  than  the  graphics. 
The  resolution  of  display  surface  was  640x800  per  eye  with  a  106  degree  field  of  view 
(FOV)  (Oculus  Rift  development  kit,  n.d.).  The  headset  was  light  (380  grams)  and 
reasonably  comfortable  allowing  for  extended  use.  Priced  at  $300.00  USD,  the  company 
sold  over  60,000  units  (Buley,  2014).  In  preparation  for  our  study,  we  thoroughly  tested 
the  headset  and  came  away  with  several  positive  opinions  about  it.  Firstly,  the  unit  was 
easy  to  set  up  and  develop  with.  Because  the  device  acted  as  an  additional  monitor  that 
relied  on  an  “extended  desktop”  metaphor,  setting  it  up  required  only  plugging  in  one 
High-Definition  Multimedia  Interface  and  one  Universal  Serial  Bus  cord.  The  device  was 
essentially  plug-and-play  as  the  drivers  utilized  by  the  software  were  all  loaded  up  on  the 
startup.  This  eliminated  the  need  for  an  external  application  dedicated  to  monitoring  the 
motion  tracking  sensors  to  be  running  on  the  host  computer.  Secondly,  the  motion 
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tracking  was  visually  impressive,  supporting  3  degrees  of  freedom  and  with  latency  of 
around  2  ms  (“Oculus  Rift  Development  Kit,”  n.d.).  For  sueh  a  low  priee  point,  the 
highly  precise  motion  tracking  of  the  unit  allowed  for  better  sense  of  presence,  had  a 
potential  to  cause  less  cybersickness,  and  offered  a  better  overall  user  (operator) 
experience.  Our  biggest  complaint  about  this  hardware  was  the  very  low  resolution  of  its 
display.  With  a  resolution  of  only  640x800,  any  seene  rendered  in  the  headset  appeared  to 
have  an  overlay  of  a  visible  pixel  grid.  As  our  simulation  would  require  not  only  a 
detailed  ship’s  bridge  but  many  augmented  HUD  elements  that  needed  to  be  easily  read, 
we  determined  that  this  headset  would  be  unsuitable  for  further  testing  and  operation. 

(2)  DK2 

The  second  hardware  release  by  Oculus  came  with  several  improvements  not  only 
in  resolution  of  its  display,  but  also  in  motion  tracking.  The  DK2  has  a  resolution  of 
960x1080  per  eye  and  although  there  still  remains  some  pixelation,  the  images  displayed 
in  the  center  portion  of  the  display  are  much  clearer.  This  model  came  with  an  upgraded 
motion  head  traeking  system  utilizing  an  infrared  camera  system.  This  resulted  in 
providing  6  DOF  of  head  motion  tracking  with  latency  comparable  to  its  predeeessor.  As 
a  user  navigates  through  a  VE,  this  type  of  tracking  allows  for  actions  like  leaning  in, 
erouehing,  and  strafing  (moving  sideways).  Due  to  a  signifieant  change  in  the  hardware 
design,  the  DK2  requires  a  software  program  installation  on  all  deviees  utilizing  the 
headset.  This  software  includes  a  configuration  utility  and  background  processes  not  seen 
in  the  first  development  kit.  Due  to  this  design  difference,  we  needed  to  introduce 
significant  changes  at  the  software  level  when  developing  the  experimental  environment 
(details  of  these  changes  are  provided  in  the  following  sections).  The  most  signifieant 
change  was  the  inability  to  continue  using  the  “extended  desktop”  mode  on  one  computer 
to  render  both  the  operator  view  and  control  view.  Instead,  a  networked  solution  of 
applications  on  two  computers  was  developed.  Our  efforts  were  also  directed  toward 
limiting  the  effects  of  cybersickness  on  the  test  subjeets  and  running  both  the  training  and 
ship  driving  simulations  at  approximately  75  frames  per  second  (FPS). 
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(3)  Unity  Integration 

With  the  new  emergence  of  consumer-grade  VR  hardware  being  brought  to  the 
market,  tools  to  create  rich  interactive  VEs  are  also  needed.  Recent  strides  in  the  indie 
video  game  industry  have  led  to  the  development  of  a  design  platform  called  Unity. 
Developed  in  2006,  the  Unity  game  engine  “was  created  with  the  vision  to  democratize 
game  development  and  level  the  playing  field  for  developers  across  the  globe”  (Unity, 
n.d.).  This  game  engine  supports  simple  scene  creation,  animation  capabilities,  real-time 
physics  rendering,  and  highly  customizable  scripting.  Combined  with  the  free  distribution 
of  nearly  all  these  tools,  this  provided  a  perfect  development  environment  for  the  study. 

Oculus  released  both  DKl  and  DK2  with  accompanying  software  development 
kits  that  utilized  Unity  prefabricated  packages.  This  allowed  for  VR  integration  into  any 
first-person  VE  with  little  more  than  one  click  to  unpack  the  provided  packages.  While 
the  AR  integration  that  would  be  utilized  in  the  study  would  require  significant 
modification  of  these  provided  scripts,  the  basic  plug  and  play  modularity  of  the  provided 
software  development  kit  (SDK)  saved  a  lot  time  during  development. 

2.  Computer  Systems 

The  final  version  of  the  software  used  a  networked  solution  in  which  the  real-time 
image  generation  (rendering)  occurred  on  one  computer  while  the  ship  movement  and 
behavior  (its  responses  to  VE  and  its  dynamic  conditions)  was  controlled  with  a 
secondary  computer. 

We  required  a  computer  with  significant  processing  power  to  render  the  3D  scene 
in  real-time  with  high  display  frame  rate  and  to  support  responsive  head  tracking.  The 
following  system  configuration  was  used  in  the  primary  computer: 

•  Processor:  Intel  Core  i7  4930k  CPU  @  3.40Ghz  (6  core) 

•  Memory:  16  GB  ISOOhz  Ram 

•  Graphics  Card:  NVIDIA  GeEorce  GTX  690 

•  Operating  System:  Windows  7 


34 


The  secondary  computer  was  not  used  to  do  any  intensive  graphical  processing 
and  thus  any  current  generation  computer  can  be  used.  In  case  of  this  study,  we  used 
Apple’s  Mac  mini  with  following  configuration: 

•  Processor:  Intel  Core  i5  2.50Ghz  (dual  core) 

•  Memory:  16  GB  1600MHz  Ram 

•  Graphics  Card:  Onboard  Intel  HD  Graphics  4000 

•  Operating  System:  OS  X  10.10.1 

The  physical  setup  and  connection  of  the  computers  is  outlined  in  Figure  2. 


Figure  2.  Device  Connection  Map 


3,  Networking  Solution 

The  final  solution  of  the  experimental  environment  required  that  both  computers 
be  connected  in  a  stand-alone  network.  Knowing  that  we  would  be  unable  to  use  an 
existing  network,  the  software  was  developed  to  use  the  following  networking  hardware. 

•  Router:  CISCO  Linksys  El 200 
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•  Cabling:  2  *  CAT5e  cables 

4.  Audio  System 

While  the  environmental  audio  was  not  a  critieal  faetor  in  the  study,  a  decision 

was  made  to  play  the  sounds  of  the  oeean  during  the  experimental  sessions  to  reproduee 
as  realistie  a  sound  stage  for  the  experimental  session  as  possible.  The  elements  of  the 
audio-setup  consisted  of  the  following  segments: 

•  Speaker:  Photive  Hydra  Bluetooth  Speaker 

•  Playback  Device:  Mac  Mini 

•  Recording:  A  recording  of  the  ocean  at  low  tide  was  used  (Roberts, 
2009).  The  audio  elip  is  10  minutes  and  11  seeonds  long  and  features 
sounds  of  the  oeean  waves  breaking  against  a  shoreline  and  seagull  ealls. 
The  recording  was  played  on  repeat  for  the  duration  of  each  experimental 
session. 

B.  SOFTWARE 

1.  Initial  Design  Decisions 

While  we  decided  early  on  that  we  would  use  VR  teehnology  in  our  user  study, 
the  aetual  configuration  of  the  experimental  environment  went  through  many  iterations 
and  design  changes.  We  started  with  major  system  requirements  and  made  our  initial 
seleetion  of  the  platform.  Firstly,  we  needed  a  eustom-made  software  applieation  in 
whieh  we  eould  utilize  the  Oeulus  Rift  or  a  similarly  designed  HMD.  Seeondly,  we 
needed  complete  aeeess  to  the  applieation ’s  souree  eode  sueh  that  we  could  add  the 
augmented  layer  for  the  HMD.  Lastly,  the  3D  environment  had  to  be  realistie  enough 
sueh  that  test  subjeets’  sense  of  presence  would  be  as  high  as  possible  with  no  major 
distraetions  to  cause  breaks  in  presenee.  We  wanted  them  to  feel  as  if  they  were  on  the 
deck  of  the  ship  and  thus  instantly  begin  eonning  without  the  need  for  extended 
aeelimation  to  the  simulated  environment.  It  was  necessary  to  ensure  that  the  experience 
of  driving  the  virtual  ship  was  close  enough  to  the  way  it  is  done  in  a  real  (operational) 
environment  sueh  that  adding  the  experimental  HUD  would  be  the  only  significant 
variable  to  the  eomparable  real-world  scenario. 


36 


The  U.S.  Navy  already  uses  VR  teehnology  in  its  shipboard  trainers.  Loeated  in 
Newport,  Rhode  Island,  “the  Conning  Officer  Virtual  Environment  (COVE)  stations 
provide  state  of  the  art  navigation  and  shiphandling  training  for  all  of  our  [U.S.  Navy] 
surface  officers.  These  trainers  can  emulate  all  of  the  U.S.  Navy’s  homeports  in  addition 
to  almost  every  routine  port  of  call  around  the  world”  (COVE,  2014,  p.  1).  The  COVE 
system  also  encapsulates  a  validated  physics  engine  for  all  primary  classes  of  ships. 
These  types  of  characteristics  drew  our  initial  intention  toward  the  COVE  system  and 
made  us  consider  using  this  system  and  its  training  program  in  our  own  testable  HUD. 
However,  we  very  quickly  determined  that  a  proprietary  nature  of  the  entire  system 
would  prohibit  making  changes  to  the  source  code  that  needed  to  be  introduced  in  support 
of  our  study. 

Our  secondary  option  was  to  go  with  the  software  that  was  already  available 
within  the  Modeling,  Virtual  Environments  and  Simulation  (MOVES)  Institute.  In  2011, 
while  attending  NFS,  Eieutenant  Commander  De  Moraes  created  an  entire  ship  driving 
trainer  utilizing  the  open  source  DeltaSD  engine  (2011).  This  training  simulation  was 
designed  to  address  “the  need  for  a  navigation  and  shiphandling  game-based  training 
system  at  naval  academies”  (de  Moraes,  2011,  p.  i).  After  testing  out  the  software  and 
taking  a  look  at  the  source  code,  we  determined  that  integration  of  the  Oculus  Rift 
hardware  with  this  code  would  be  a  non-trivial  task. 

As  discussed  in  the  Chapter  III,  the  Oculus  Rift  was  designed  to  be  supported  by 
contemporary  game  engines  such  as  Unity  or  Unreal  Engine.  At  the  time,  there  were  no 
integration  packages  available  for  the  DeltaSD  engine.  We  determined  that,  given  the 
alternatives,  the  most  effective  way  to  create  a  usable  experimental  environment  for  the 
study  would  be  to  build  a  new  custom  application  in  a  Unity  environment.  Having  made 
this  determination,  we  began  the  development  of  a  VR  ship  driving  simulation  in  March 
of2014. 
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2,  Simulation  Development 

a.  Unity 

Unity  is  modular  game  development  software  suite  that  allows  users  to  take 

nearly  any  format  of  3D  model  and  create  rich  VEs.  With  an  active  community  of  over 
2,000,000  registered  developers,  there  is  a  large  number  of  tutorials,  example  code,  and 
forums  available  online,  making  it  an  easy  platform  to  work  with.  We  used  a  combination 
of  publicly  available  3D  models  as  well  as  3D  models  already  created  by  the  Visual 
Simulation  and  Game-Based  Technology  group  at  NFS  for  objects  in  the  NAVHUD 
simulation.  The  scripts  utilized  in  the  study  were  all  written  in  C#  programing  language. 
Most  of  the  issues  detailed  in  this  chapter  were  tested  and  created  in  isolation,  and  then 
subsequently  integrated  into  the  main  simulation. 

b.  Initial  Design 

A  need  for  two  independent  views,  an  operator  view  and  a  controller 
(experimenter)  view,  became  evident  soon  after  the  coding  for  the  simulation  began.  The 
operator  view  represented  the  first-person  perspective  from  the  ship’s  bridge  displayed 
inside  the  Oculus  Rift.  This  showed  what  the  conning  officer  would  see  from  the  bridge. 
Additionally,  the  controller  (experimenter)  view  would  be  a  two-dimensional  graphical 
user  interface  (GUI)  featuring  a  series  of  ship  driving  controls  and  raw  sensor  data 
outputs  (readings).  These  panels  would  not  only  allow  the  “helmsman”  to  drive  the  ship, 
but  they  would  also  be  used  by  the  “navigation  evaluator”  to  construct  verbal  reports  for 
the  conning  officer. 

In  the  first  stage,  we  used  the  DKI  to  mirror  a  single  monitor  for  a  conning  officer 
(the  subject  in  the  study).  A  dual  monitor  design  (i.e.,  extended  desktop  metaphor) 
supported  both  the  subject  (conning  officer)  and  two  experimenters  (helmsman  and 
navigation  officer).  This  design  presented  the  visual  output  generated  by  the  simulation 
over  two  monitors;  the  first  being  dedicated  to  the  operator’s  view  and  the  second  being 
dedicated  to  the  controller  view.  The  monitor  dedicated  to  the  operator  was  duplicated  on 
the  Oculus  Rift — this  way  both  the  operator  (subject/conning  officer)  and  the  controllers 
(experimenters/helmsman  and  navigation  officer)  were  able  to  see  the  same  visuals.  The 
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initial  stage  of  code  development  that  lasted  several  months  used  this  method  -  all  needed 
data  were  properly  displayed  within  one  running  executable  application.  While  simplistic 
in  its  overall  structure  this  design  satisfied  the  basic  needs  of  the  study.  Although  we 
realized  that  the  low  resolution  of  the  DKl  made  it  unusable  in  the  study,  we  planned  to 
transition  to  the  DK2  after  its  scheduled  release  date  in  July  of  2014  (Oculus  Team, 
2014).  Being  that  both  units  were  produced  by  the  same  company  and  bundled  with 
similar  Unity  integration  packages,  we  assumed  that  any  software  developed  using  DKl 
concepts  would  be  forward  compatible  to  the  DK2. 

After  its  arrival  in  late  July,  the  DK2  was  thoroughly  tested  and  significant 
differences  in  the  hardware  implementation  were  discovered.  Whereas  the  DKl  primarily 
acted  as  an  additional  monitor  on  extended  desktop,  the  DK2  was  much  more 
sophisticated  and  even  required  a  software  executable  to  be  running  in  the  background  to 
manage  the  device.  While  this  was  beneficial  to  many  programs  that  would  utilize  the 
Oculus  Rift  hardware,  it  proved  to  be  detrimental  to  the  initial  system  architecture  that  we 
had  created  for  the  study.  The  benefit  that  this  new  software  provides  is  that  it  no  longer 
requires  a  screen  to  be  mirrored  for  the  hardware  to  work  but  instead;  the  Oculus 
applications  are  ran  as  background  tasks  and  still  render  perfectly  fine  in  the  HMD. 
Given  the  impossibility  of  running  the  shipboard  VR  environment  that  had  been 
developed  thus  far  within  the  DK2,  (since  there  was  no  support  for  extended  desktop  in 
configuration  needed  for  our  study)  we  determined  that  a  networked  solution  was  the 
only  feasible  option. 

c.  Networked  Solution 

Moving  to  the  networked  solution  required  by  the  DK2  was  a  difficult  task.  After 
several  methods  of  splitting  up  the  application  and  testing  it  (special  attention  was  paid  to 
operator’s  visual  experiences  and  usability  during  the  actual  sessions),  we  determined 
that  the  best  course  of  action  was  to  have  the  operator’s  view  of  the  simulation  acting  as  a 
host  and  the  controller’s  view  of  the  ship  to  be  a  client  application.  The  primary  reason 
for  this  specific  host-client  configuration  was  that  the  Unity  networking  utilities  caused 
slight  delays  in  client  applications.  It  was  critical  that  the  scenes  rendered  in  the  HMD 
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and  related  visual  flow  be  as  smooth  as  possible  to  minimize  if  not  eompletely  diminish 
the  symptoms  of  eybersickness;  this  was  aehieved  by  using  the  applieation  responsible 
for  the  operator  view  to  be  the  host  applieation. 

In  the  final  design  of  the  simulation,  both  the  host  and  elient  start  loading  up  an 
identieal  seene  but  from  different  perspeetives.  This  seene  represents  a  ship  that  is 
stationary  in  the  water  plaeed  at  the  start  of  the  ehannel.  The  host  applieation  depiets  the 
first-person  view  from  inside  the  bridge  whereas  the  elient  applieation  presents  a  third- 
person  perspeetive  following  the  ship  from  a  set  distanee  above  and  behind  the  ship.  The 
host  applieation  integrates  the  AR  graphics  and  the  client  is  where  the  ship  controls  and 
navigation  reports  were  placed  for  the  benefit  of  the  controller  (experimenter). 

The  process  of  adding  more  3D  objects,  scripts,  and  physics  to  the  simulation 
increased  the  lag  in  the  HMD.  The  overarching  goal  for  the  study  was  to  design  a  system 
that  looked  and  felt  real  enough  to  complete  the  task  of  conning  a  ship  out  of  the  channel 
while  simultaneously  not  causing  the  user  discomfort  due  to  eybersickness.  It  was 
therefore  critical  that  the  execution  of  the  host  application  contained  as  little  overhead  as 
possible.  In  order  to  do  this,  we  had  the  host  and  client  computer  run  the  same  simulation 
while  the  client  sent  small  remote  procedure  calls  (RPC)  to  itself  and  the  host.  These 
RPCs  included  information  about  the  engine  and  rudder  orders,  shifts  of  the  navigation 
waypoints,  and  bridge  movement  commands. 

The  last  segment  we  developed  to  help  control  this  networked  solution  was  a  GUI 
that  was  used  on  client  computer  to  initiate  the  connection  between  the  two  devices.  This 
GUI  comprised  of  a  textbox  for  subject  identifier,  selectors  for  the  experimental 
condition  to  be  used  in  a  given  session,  and  a  textbox  for  the  local  Internet  Protocol  (IP) 
address.  Once  submitted,  the  two  programs  would  create  a  connection  allowing  the  RPCs 
to  be  passed  back  and  forth.  As  a  means  of  making  things  simpler,  the  IP  address  box  was 
populated  with  the  local  IP  address  of  the  client  application.  Because  both  devices  would 
always  be  on  the  same  router,  only  the  last  three  digits  needed  to  be  changed  to  specify 
the  host. 
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d. 


3D  Ship  Model 


One  of  the  design  priorities  was  seleeting  highly  detailed  3D  model  of  a  U.S. 
Navy  warship.  The  ideal  vessel  for  testing  was  a  eruiser/destroyer  sized  warship;  our 
guidanee  in  this  deeision  was  the  assumption  that  more  surface  warfare  officers,  our 
potential  subjects  in  user  study,  would  have  the  knowledge  and  skills  to  drive  this  size  of 
ship  than  our  other  options.  The  best  model  available  to  us  was  a  model  of  a  Littoral 
Combat  Ship  (LCS)  in  the  MOVES  Institute  repository  of  3D  models.  The  most 
significant  issue  with  the  model  however  was  a  lack  of  geometry  description  for  internal 
rooms.  That  meant  that  we  had  to  design  geometry  of  the  bridge  and  insert  it  into  the  3D 
model  of  the  ship.  The  image  in  Figure  3  shows  the  outer  hull  of  the  ship  and  the  image 
in  Figure  4  shows  the  internal  design  of  the  bridge;  the  entire  model  of  the  ship  used 
60,000  polygons.  Photos  of  the  real  ship  and  bridge  were  used  as  a  reference  to  ensure 
that  the  design  replicated  a  “look  and  a  feel”  of  the  real-world  FCS  bridge  as  much  as 
possible. 


Figure  3.  FCS  Model  (60,000  Polygons) 
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Figure  4.  LCS  Bridge 


e.  Environment 

The  next  step  toward  creating  a  highly  realistic  scene  was  the  acquisition  of  a  3D 
model  that  described  the  terrain  and  water  environment,  both  needed  to  simulate  the  ship 
transit.  The  initial  attempts  focused  on  using  a  scaled  model  of  San  Diego  Bay;  we 
imported  the  terrain  height  data  and  overlaid  Google  Map  aerial  imagery.  This  created  a 
very  realistic  scene  that  depicted  a  locale  known  to  naval  officers.  As  the  design  of  study 
progressed,  we  recognized  that  in  order  to  get  usable  data  we  would  need  to  make  certain 
aspects  of  the  transit  artificial.  The  San  Diego  transit  had  significantly  long  tracks,  and 
skilled  conning  officers  would  have  no  difficulty  gaining  and  maintaining  track  over 
those  distances;  this  locale  would  not  challenge  their  conning  skills  and  thus  not  provide 
us  with  the  best  basis  to  test  our  research  hypothesis.  Therefore,  we  decided  that  a  far 
better  solution  would  be  to  have  a  series  of  shorter  legs  with  distributed  winds  and 
currents  in  them  that  would  force  conning  officers  to  take  significant  action  to  conduct 
their  basic  task  -  to  drive  the  center  of  the  track  during  each  leg  of  the  transit. 

The  creation  of  an  artificial  environment  (3D  model  of  the  terrain  and  water) 
brought  about  several  advantages.  Firstly,  it  allowed  for  a  consistency  of  length  for  each 
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leg.  We  ended  up  ehoosing  1,500  yards  for  eaeh  transit  leg.  With  1,500  yards  per  leg,  we 
were  able  set  the  transit  speed  at  15  knots  (a  reasonable  speed  for  an  outbound  transit) 
and  have  the  turns  between  the  adjoining  legs  happen  every  three  minutes.  In  the  study, 
we  needed  eaeh  partieipant  to  drive  the  same  transit  three  times  (once  in  each 
experimental  condition).  Additionally,  it  was  necessary  to  optimize  the  total  time  of  each 
session  so  that  the  entire  experience  for  one  subject  did  not  last  too  long.  At  three  minutes 
per  leg  and  a  desired  total  time  of  15  minutes  per  condition,  we  ended  up  with  five  legs 
per  user  per  session  (condition).  This  allowed  each  run  to  produce  statistically  significant 
data.  The  second  reason  for  a  consistent  leg  length  was  the  ability  to  guarantee  that  each 
leg  would  provide  ample  time  for  three  full  navigation  reports,  the  significance  of  which 
will  be  covered  in  greater  detail  in  Chapter  V.  Finally,  the  consistent  leg  length  allowed 
the  conning  officers  to  focus  on  their  task  of  staying  on  track  while  knowing  that  turns 
would  occur  at  regular  intervals.  The  “three  minute  rule”  is  a  commonly  used  tool  for 
those  who  drive  ships  as  a  part  of  their  regular  professional  duties.  Stated  simply,  in  three 
minutes  a  ship  will  travel  its  current  speed  in  knots  multiplied  by  100  yards.  In  this  case, 
with  a  constant  speed  of  approximately  15  knots,  the  conning  officer  would  know  that 
1 ,000  yards  until  turn  would  have  given  them  two  more  minutes  to  regain  track,  and  so 
forth. 

The  scene  developer  tools  in  Unity  software  package  were  used  to  craft  a 
fictitious  channel  that  consisted  of  five  legs  of  1,500  yards  each.  The  image  in  Figure  5 
shows  six  legs,  the  first  being  a  starting  leg  that  required  no  commands  from  the  conning 
officer.  This  initial  leg  also  gave  the  ship  opportunity  to  come  to  speed  before  the 
conning  officer  needed  to  take  action.  We  also  included  several  lifelike  3D  models  of 
other  stationary  vessels  as  well  as  buildings,  all  aimed  at  increasing  a  sense  of  presence  of 
the  subjects  in  our  study.  We  acquired  all  of  these  models  either  through  the  MOVES 
Institute  or  through  free  online  model  sharing  websites  (e.g.,  www.turbosquid.com  and 
www.tf3dm.com).  The  image  in  Figure  6  presents  a  more  realistic  depiction  of  what  the 
conning  officers  would  expect  to  see  during  their  transit. 
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ISLA  VERDE 


270* 


300* 

1500  yds 


1500  yds 


270*  ^ 

1000  yds 


Figure  5.  Map  of  Isla  Verde  with  Traek  Details 


Figure  6.  Isla  Verde  with  3D  Models 
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f.  Ship  Physics 

(1)  Acquisition 

Besides  finding  an  adequate  ship  model,  the  physies  engine  for  moving  the  ship 
through  the  environment  was  the  most  important  requirement  to  aeeomplish  while 
reereating  the  elements  of  environment  that  conning  officers  would  respond  to.  If  the  ship 
handled  signffieantly  different  from  the  expeetations  of  the  subjeets  (eonning  offieers) 
then  the  learning  eurve  for  eaeh  session  would  render  any  eolleeted  data  useless. 
Modeling  the  oeean  physies  systems  is  a  non-trivial  endeavor  and  requires  a  signifieant 
amount  of  expertise;  therefore  eonsiderable  efforts  were  invested  to  find  and  ineorporate 
existing  high-quality  maritime  physies  model  into  our  applieation.  During  an  exploratory 
visit  to  the  labs  of  BlueShark  at  the  Institute  for  Creative  Teehnologies  at  the  University 
of  Southern  California,  we  diseovered  that  they  had  already  designed  sueh  a  system  for 
their  AR  demonstration  model.  Blueshark  is  a  team  of  researehers  and  engineers 
sponsored  by  the  Offiee  of  Naval  Researeh  (“E2C2  Team,”  2015).  They  speeialize  in  the 
researeh  and  development  of  augmented  and  mixed  reality  systems  tailored  for  military 
applieation.  The  system  that  we  wished  to  partially  emulate  was  a  ship  driving  experienee 
that  integrated  a  full  immersion  HMD  with  infrared  traeking  gloves.  While  not  fully 
based  on  genuine  physies,  the  system  paralleled  the  real-world  motions  of  the  ship  quite 
accurately.  The  BlueShark  team  was  willing  to  share  their  code  that  governed  the  physics 
of  the  ship  and  that  beeame  the  baseline  of  ship  movement  through  the  water. 

(2)  Integration 

The  shared  eode  represented  an  engine  with  funetion  ealls,  and  thus  it  was 
neeessary  to  build  a  custom  driving  interface  that  would  allow  the  operator  (helmsman) 
to  easily  eontrol  the  engines  and  rudder  of  the  ship.  The  initial  ereation  of  the  driving 
eontrols  was  simple;  however,  signifieant  ehanges  had  to  be  added  to  support  the 
networked  solution  of  the  system.  We  deeided  that,  using  RPC  functions,  the  elient 
applieation  would  send  orders  to  both  itself  and  the  host  simultaneously.  These  orders 
were  either  propulsion  values  or  rudder  angles  (details  on  the  final  design  of  the  driving 
interface  can  be  found  in  the  Controller  HUD  seetion  of  this  ehapter.) 
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The  system  integration  involved  more  than  simply  adjusting  the  inputs  to  the 
physics  engine.  Because  the  Unity  game  engine  allows  the  models  to  be  scaled  arbitrarily 
and  the  physics  engine  we  were  using  came  with  no  guarantee  of  being  validated,  there 
was  no  immediate  way  to  tell  if  our  ship  was  really  traveling  at  its  reported  speed  or  not. 
To  validate  that  they  performed  correctly,  we  performed  many  tests  with  3D  models  and 
integrated  ship  behavior  (physics)  to  check  the  scale,  speed,  and  turning  radius  values  of 
the  vessel.  Using  this  approach,  we  were  able  to  verify  the  data  we  need  to  accurately 
report  on  parameters  such  as  speed  over  ground  (SOG),  distance  off  track,  distance  to 
next  turn,  and  others. 

(3)  Optimization 

The  trial  runs  of  conning  a  demonstration  course  helped  us  determine  that, 
without  the  inclusion  of  external  forces  which  would  additionally  influence  the  motion  of 
the  ship,  the  task  in  our  study  would  be  too  easy  for  skilled  conning  officers.  The 
measurement  we  were  interested  to  collect  and  record  was  distance  off  the  track 
throughout  the  transit.  In  a  situation  with  perfect  environmental  conditions  the  offset 
from  the  main  track  would  be  minimal  in  each  session,  providing  us  with  no  significant 
results  and  no  basis  to  test  our  research  hypothesis.  To  increase  the  difficulty  of  the 
transit,  set  and  drift  forces  were  added  to  the  simulation. 

Set  and  drift  are  forces  enacted  upon  vessels  from  both  winds  and  sea  (ocean) 
currents.  Set  refers  to  the  direction  to  which  the  ship  is  being  pushed.  Drift  is  the  speed  at 
which  these  forces  are  pushing  the  vessel.  Since  no  such  system  was  built  into  the  physics 
engine  we  had  adopted,  the  appropriate  model  of  those  forces  needed  to  be  added  to  the 
overall  engine.  Once  this  model  was  completed,  the  controls  to  physically  adjust  the 
forces  of  set  and  drift  were  integrated  into  the  waypoint  system.  The  waypoint  system  - 
covered  further  on  in  the  chapter — is  the  primary  scripting  agent  for  our  experiment.  This 
system  dictates  the  course  track,  external  forces  (set  and  drift),  and  records  performance 
data. 
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(4)  Course  Over  Ground 

During  the  final  stages  of  testing,  we  paid  special  attention  to  the  accuracy  of  the 
reporting  mechanism  for  the  ship’s  course  over  ground  (COG).  COG  is  an  important 
report  to  understand.  Ships  do  not  always  move  in  the  direction  they  are  headed.  The 
influences  of  wind  and  current  on  the  ship  act  as  independent  vectors  that,  when  added 
together  with  the  ships  own  propulsion  vector,  can  lead  to  a  direction  of  motion  that  is 
different  than  the  direction  of  the  ship’s  heading.  When  this  effect  is  signiflcant,  and  the 
ship  must  point  its  bow  in  a  different  direction  than  the  course  it  wants  to  make  in  the 
water,  this  is  called  “crabbing”  (Figure  7).  From  the  perspective  of  a  conning  offlcer, 
when  he  orders  the  helmsman  to  steer  a  certain  course  and  the  ship’s  COG  differs 
signiflcantly  from  the  ordered  course,  that  is  a  clear  indication  of  how  set  and  drift  are 
affecting  the  vessel.  Alternatively,  if  the  conning  offlcer  knows  what  the  set  and  drift  are, 
he  can  anticipate  their  effects  and  order  a  course  that  compensates  for  these  elements 
allowing  the  ship  to  maintain  track.  When  the  addition  of  the  set  and  drift  mechanics  flrst 
occurred,  the  effects  it  would  have  on  the  COG  report  were  overlooked.  The  final  version 
of  the  reporting  function  took  into  the  account  this  phenomenon. 
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Figure  7.  Depiction  of  a  Ship  That  Is  Crabbing 


g.  First-Person  Movement 

One  of  the  goals  for  this  project  was  to  utilize  emerging  VR  hardware  systems 
capable  of  supporting  user  interactions  that  had  never  been  employed  in  the  realm  of  ship 
driving  simulations  before.  A  task  that  had  never  been  tackled  so  far  was  to  allow  users  to 
actually  walk,  i.e.,  freely  navigate  around  the  bridge.  The  closest  simulation  that  has 
allowed  for  this  was  the  Navy’s  Full  Mission  Bridge  (FMB)  Simulator  (“FMB,”  2014). 
This  system  is  a  physical  mock-up  of  the  bridge  which  has  large  screens  placed  in  a  semi¬ 
circle  around  it.  The  trainees  can  move  about  freely  about  the  bridge  mock-up.  A  clear 
advantage  of  this  system  is  that  it  provides  a  highly  realistic  multiuser  experience  which 
trains  many  roles  simultaneously.  The  most  significant  disadvantage  of  this  system  is  the 
extremely  high  cost  to  build  and  run  such  a  trainer.  Excluding  the  FMB,  most  other 
simulators  place  the  conning  officer  on  the  bridge  and  allow  him  to  move  to  the  bridge 
wings  via  a  warping  movement  that  can  be  best  explained  as  a  teleporting  capability.  The 


systems  usually  incorporate  a  series  of  viewpoints,  inside  and  outside  the  ship,  to  which 
the  conning  officer  can  ask  to  change  his  current  view.  One  of  the  goals  of  the  NAVHUD 
project  was  to  build  a  system  in  which  the  users  had  a  full  control  of  their  movement  on 
the  bridge. 

We  started  with  a  goal  of  full  human  locomotion.  The  Virtuix  Omni,  in  many 
aspects,  mimics  the  effort  taken  by  the  Oculus  team  in  their  attempts  to  bring  a 
commercial  VR  to  the  masses  of  potential  users  (Figure  8).  The  Omni  brings  the 
capabilities  of  a  multidirectional  treadmill  by  keeping  a  user  in  place  while  he  moves 
around  (locomotes)  in  the  space.  The  user  is  strapped  around  his  waist  to  the  support  ring, 
and  he  locomotes  inside  a  concave,  spherical  platform,  while  the  curvature  of  the  surface 
as  well  as  the  weight  of  the  user  brings  him  “back”  toward  the  center  of  the  platform,  i.e., 
the  place  where  he  started  (tracking  sensors  are  attached  to  the  user’s  shoes.)  Users  can 
put  on  a  HMD  and  see  their  physical  movements  replicated  in  the  simulation.  This 
product  was  initially  scheduled  for  its  release  in  August  of  2014,  and  was  seen  as  a  viable 
way  of  allowing  subjects  to  physically  control  their  own  movements  while  navigating  on 
3D  representation  of  the  bridge.  Scheduling  delays  resulted  in  the  product  not  being 
released  in  time  for  testing,  and  therefore  a  different  navigation  method  needed  to  be 
adopted  and  developed  for  the  study. 


Figure  8.  OMNI  Hardware  (photo  used  with  permission  from  Virtuix,  n.d.) 
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A  special  amount  of  attention  was  invested  in  devising  a  correct  transformation 
mechanism  for  each  object  in  the  scene.  Most  specifically  the  movement  for  the  conning 
officer  on  the  bridge  needed  to  account  for  the  vessel’s  own  movement.  The  challenge 
was  allowing  the  character  to  move  independently  of  the  ship,  while  simultaneously 
applying  the  ship’s  own  movement  vectors  to  the  character.  With  the  help  of  the  MOVES 
Visual  Simulation  and  Game-Based  Technology  group,  an  adequate  solution  was 
devised.  The  key  to  the  solution  was  to  attach  the  character  to  the  ship  so  that  ship 
position  updates  would  trickle  down  to  the  character.  The  second  step  was  to  write  code 
into  the  LateUpdate()  function  so  that  these  transformations  would  not  collide  with  the 
ship’s  own  transformations.  This  resulted  in  smooth  movement  of  the  character. 

Since  the  acquisition  of  OMNI  hardware  did  not  materialize,  a  secondary  option 
for  character  movement  was  sought.  An  initial  suggestion  was  to  use  controllers  such  as 
an  Xbox360  game  controller  to  give  subjects  direct  control  over  their  movement  i.e., 
navigation  through  the  3D  scene.  We  decided  this  solution  was  inadequate  as  it  would 
require  additional  training  for  subjects  not  familiar  with  game  controllers.  We  also 
wanted  to  avoid  an  unneeded  distraction  in  the  basic  task  of  ship  driving. 

Instead,  we  implemented  a  method  often  utilized  in  arcade  first-person  shooters 
called  “rail  shooters.”  This  method  uses  several  predefined  viewpoints  in  any  scene  and 
moves  the  user  automatically  and  smoothly  between  them.  This  particular  solution  was 
determined  to  be  adequate  based  on  our  knowledge  of  domain  space  and  the  behaviors 
exhibited  by  the  operators  -  conning  officers  would  only  be  likely  to  place  themselves  in 
one  of  a  few  typical  places  on  the  bridge  from  which  they  usually  observe  the  outside 
environment  and  conduct  their  tasks,  and  so  there  was  less  need  to  allow  complete 
freedom  of  movement  within  the  bridge.  The  positions  in  this  study  included  the 
“centerline”  (middle  of  the  bridge)  and  the  left/right  bridge  wings  respectively.  The  final 
results  allowed  the  conning  officers  to  verbally  ask  the  experimenter  to  move  him  to  one 
of  four  possible  positions  on  the  bridge:  (1)  the  left  bridge  wing,  (2)  the  right  bridge 
wing,  (3)  the  left-center  bridge,  or  (4)  the  right-center  bridge.  The  reason  for  the 
separation  of  the  centerline  bridge  positions  is  because  our  model  is  of  a  ECS  class  ship 
where  the  centerline  is  blocked  by  a  support  beam  and  electrical  control  equipment. 
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h.  Water  Rendering 

Another  important  element  of  a  realistic  3D  shipboard  environment  is  a  very  good 
simulation  of  water.  Similar  to  our  search  for  3D  models  of  ships  and  the  surrounding 
environment,  a  tool  to  realistically  render  water  was  sought  out  early  in  the  design  of  the 
simulation.  Physical  cues  such  as  the  direction  and  speed  of  the  current,  depth  of  the 
water,  and  overall  sea  state  can  be  derived  from  the  look  of  the  water  surface.  These  cues 
are  essential  inputs  to  the  conning  officer,  who  will  in  turn  give  commands  that  account 
for  the  additional  forces  acting  on  the  ship.  If  the  water  that  was  rendered  was  perceived 
to  be  too  unrealistic,  we  supposed  it  would  have  been  a  source  of  distraction  for  the  test 
subjects.  An  initial  investigation  led  to  the  choice  of  an  engine  called  Triton  Oceans  by 
developer  Sundog.  While  the  software  is  a  popular  choice  for  use  by  military  simulation 
companies,  the  primary  reason  for  use  was  their  excellent  water  rendering  samples 
(Figure  9).  The  system  not  only  renders  realistic  water,  it  also  adds  a  simulation  of  ocean 
currents,  buoyancy,  and  winds  (the  effects  we  judged  as  necessary  in  our  simulation).  The 
tools  themselves  are  designed  as  packages  that  can  be  easily  imported  into  the  Unity 
game  developer  and  we  hoped  that  the  final  integration  would  be  simple.  After 
attempting  to  integrate  the  ocean  packages  into  our  simulation,  a  significant  issue  was 
discovered.  The  Oculus  Rift  uses  two  separate  camera  positions  in  the  scene  and  then 
applies  multiple  effects  during  the  rendering  process.  This  process  allows  for  the 
stereoscopic  vision,  i.e.,  it  enables  stereoscopic  depth  cue  that  gives  the  user  an 
impression  of  being  in  a  3D  environment.  Due  to  the  complex  rendering  techniques  used, 
the  final  rendering  of  the  Triton  Ocean  assets  processing  showed  numerous  artifacts.  This 
issue  was  known  by  the  team  at  Sundog  Software  and  has  since  been  fixed,  but  that  did 
not  occur  in  time  for  our  testing  (Dykes,  2014). 
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Figure  9.  Triton  Ocean  Example  (used  with  permission  from  Sundog 

Software,  n.d.) 


Several  attempts  were  made  to  work  with  the  team  members  of  Sundog  and 
BlueShark;  however,  no  solution  was  found  for  correct  functioning  of  the  Triton  Ocean 
water  rendering  in  our  application.  Instead,  we  decided  to  use  the  built-in  water  rendering 
tools  provided  by  Unity.  The  image  in  Figure  10  shows  a  water  scene  rendered  by  the 
professional  version  of  Unity.  While  it  is  possible  to  apply  physics  and  waves  to  this 
ocean  package,  scheduling  timelines  prevented  the  design  team  from  manipulating  the 
water  to  these  extremes. 
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Figure  10.  An  Example  of  Unity  Water  Rendering  Used  in  NAVFIUD  Study 

i.  Ship  Waypoint  System 

The  task  of  ereating  a  ship  waypoint  system  consisted  of  three  main  goals.  Firstly, 
the  waypoints  had  to  be  populated  using  a  standard  method  that  allows  for 
predetermination  of  the  distance  and  angle  between  them.  Secondly,  the  ship  had  to  be 
tracked  along  the  paths  between  the  waypoints.  Lastly,  there  had  to  be  a  standardized 
method  of  calculating  when  to  have  the  tracking  system  shift  to  the  next  leg  correctly. 
The  concepts  of  each  of  these  goals  were  fairly  simple;  however,  the  actual  coding  of 
each  element  was  non-trivial. 

The  waypoint  system  was  designed  to  be  the  backbone  of  the  entire  data 
collection  system.  We  created  a  track  that  forced  the  subjects  to  utilize  all  their  conning 
officer  skills  while  simultaneously  providing  good  points  of  reference  for  data  collection. 
The  general  concept  was  that  the  waypoints  needed  to  be  placed  on  the  map  in  such  a 
way  that  the  tracks  created  between  those  waypoints  were  all  standardized.  Additionally, 
the  waypoints  needed  to  take  into  account  the  precise  turn  angles  between  them.  The  final 
solution  incorporated  manual  calculations  for  the  positions  using  vector  mathematics,  and 
then  placing  the  waypoints  in  the  scene  accordingly.  The  angles  and  distances  were  used 
as  parameters  in  a  formula  that  used  an  origin  point  (3D  Vector)  as  input  and  produced  a 
series  of  coordinates  for  the  waypoints.  The  final  result  was  a  highly  precise  course  with 
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exact  turn  angles  and  distances.  As  shown  in  Figure  5,  the  final  version  of  the  course 
included  five  turns  averaging  64  degrees  with  three  turns  to  starboard  and  two  turns  to 
port.  This  was  determined  to  be  an  adequate  challenge  to  the  participants  while  still  being 
a  realistic  representation  of  an  actual  transit. 

The  second  challenge  was  to  build  a  tracking  mechanism  for  the  course.  The 
calculation  of  how  far  along  the  track  the  ship  had  traveled  was  a  simple  vector 
subtraction  problem,  but  more  complex  step  was  to  calculate  the  data  that  indicated  the 
distance  away  from  the  track.  Using  vector  mathematics,  we  implemented  a  system  that 
reported  the  exact  distance  left  or  right  of  track.  An  elegant  coding  solution  for  this 
method  was  found  on  Unity’s  own  message  boards,  and  it  was  deployed  in  our  code 
(Mcdroid,  2011).  The  basic  concept  was  to  take  the  difference  between  the  current 
location  of  the  ship  and  the  last  waypoint  hit.  A  cross  product  of  the  direction  of  the  track 
and  the  result  of  that  subtraction  gives  a  new  vector.  This  final  vector’s  “y”  component  is 
the  distance  off  the  track  (if  it  was  negative  it  was  right  of  track,  if  it  was  positive  it  was 
determined  as  left  of  track).  The  simulation  recorded  the  distance  off  track  once  every  10 
yards  as  the  ship  approached  the  next  turn.  To  exclude  erroneous  data  collected  during 
the  turns  themselves,  the  system  started  reporting  this  data  at  1,350  yards  till  the  next  turn 
(note;  each  leg  was  1,500  yards  long). 

The  final  function  of  the  waypoint  system  provided  the  capability  to  adequately 
judge  when  a  turn  was  to  be  made.  The  initial  design  relied  on  placing  a  theoretical  circle 
around  the  waypoints  themselves  and  having  the  system  shift  waypoints  when  the  ship 
entered  the  area  of  these  circles.  During  testing  this  was  discovered  to  be  an  inaccurate 
way  of  determining  when  to  shift  the  waypoints.  The  problem  was  twofold;  firstly,  if  the 
ship  was  significantly  left  or  right  of  track  then  it  would  never  enter  the  circle  and  thusly 
the  next  course  would  never  be  triggered.  Secondly,  the  method  of  reporting  distance  to 
the  next  leg  was  calculated  incorrectly.  The  initial  calculation  took  the  next  waypoint 
location  and  subtracted  the  ship’s  own  position.  As  ships  that  are  left  or  right  of  track 
need  to  compensate  for  their  turns  using  advance  and  transfer  tables,  the  measurement  of 
distance  to  waypoint  is  not  nearly  as  useful  as  a  measurement  of  distance  from  the  next 
leg.  This  effect  is  highlighted  in  the  image  in  Figure  11;  the  red  line  represents  the  initial 
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execution  and  the  green  line  represents  the  correct  method  to  measure  distance  to  turn.  To 
calculate  the  length  of  the  green  line,  a  ray  from  the  ship’s  position  was  extended  in  the 
direction  parallel  to  the  current  leg  direction.  The  distance  between  the  ship  and  the 
intersection  of  this  line  with  the  next  leg  is  what  was  used  in  the  final  simulation. 
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Figure  1 1 .  Turning  Mechanism 


In  order  to  accurately  determine  when  the  ship  should  turn,  we  had  to  take 
advance  and  transfer  into  consideration.  The  image  in  Figure  12  presents  advance  and 
transfer.  Ships  do  not  turn  instantly,  thus  calculations  are  used  to  precisely  determine 
when  the  ship  must  begin  a  turn  so  that  they  end  the  turn  in  the  center  of  the  next  leg’s 
track.  The  variables  taken  into  account  are  the  ship’s  speed,  the  angle  of  turn,  and  the 
ship’s  handling  characteristics.  A  number  of  tests  were  run  to  help  us  build  accurate 
advance  and  transfer  tables  for  the  vessel  used  in  our  simulation.  These  tables  were  then 
applied  to  the  recommended  distance-to-turn  value  conveyed  to  the  conning  officers.  This 
was  done  so  that  no  matter  how  far  off  (left  or  right  of)  course  they  were  at  the  end  of  a 
leg’s  track,  the  ship  would  be  positioned  near  the  center  of  the  track  at  the  beginning  of 
the  next  leg. 
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Figure  12.  Advance  and  Transfer 

A  final  function  built  into  the  waypoint  system  was  the  inclusion  of  set  and  drift 
values  for  each  turn.  Although  the  mechanics  for  implementing  set  and  drift  were 
executed  in  the  physics  system,  the  waypoint  system  was  in  charge  of  updating  the  set 
and  drift  for  each  course  leg.  One  thing  that  was  not  programmed  into  the  system  was  the 
accountability  for  set  and  drift  in  the  turn  recommendations.  Current  navigation  systems 
can  use  known  set  and  drift  conditions  to  provide  more  accurate  recommendations  of 
when  to  make  a  turn  and  at  what  course  conning  officers  should  drive  to  compensate  for 
significant  wind  and  sea  (ocean)  currents.  We  purposefully  did  not  include  this 
calculation  in  our  system  to  require  the  conning  officers  to  actively  work  and  compensate 
for  the  set  and  drift  themselves.  This  represented  a  reasonable  expectation  of  any 
experienced  conning  officer,  and  so  the  task  was  not  seen  as  overburdening. 

j.  Auto-Helmsman 

During  the  initial  testing  of  the  system,  we  concluded  that  the  manual  controlled 
rudder  was  not  only  tiresome,  but  also  a  variable  that  needed  to  be  controlled  tightly. 
When  a  conning  officer  gives  a  command  to  come  to  a  certain  course,  it  is  the  job  of  a 
highly  qualified  master  helmsman  to  take  the  input  and  manually  steer  the  ordered 
course.  Everything  from  the  current  underneath  the  rudder  to  the  responsivity  of  the 
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vessel  can  be  contributing  factors  of  how  much  rudder  the  helmsman  uses  to  come  to  an 
ordered  course.  It  is  the  job  of  the  helmsman  to  quickly  come  to  the  correct  course 
without  significantly  oversteering  (the  act  of  driving  past  an  ordered  course). 
Oversteering  is  a  common  occurrence  for  less  experience  helmsman  or  when  a  ship  is 
sailing  in  obverse  conditions.  Mistakes  of  this  nature  can  affect  the  correctness  of 
performance  data  associated  with  the  conning  officer. 

In  order  to  accurately  and  consistently  portray  the  actions  of  a  genuine  helmsman, 
and  avoid  undue  influence  over  the  performance  data  for  conning  officers,  we  desired  an 
auto-steering  system.  Research  into  existing  systems  helped  us  to  identify  a  proportional 
integral  derivative  (PID)  controller  (Minorsky,  1922),  which  is  often  used  in  engineering 
systems  (Bennett,  1984).  The  main  concept  is  that  the  controller,  when  given  an  ordered 
course  and  a  maximum  allowed  rudder  angle,  will  adjust  the  rudder  such  that  the  ship 
quickly  comes  to  the  desired  course.  Using  an  internal  feedback  loop  it  can  monitor  the 
ship’s  turning  speed  and  adjust  the  rudder  accordingly.  Thus,  if  the  ship  was  turning  too 
quickly  and  would  likely  oversteer,  the  PID  controller  will  apply  the  opposite  direction  of 
rudder  to  counteract  these  forces.  This  behavior  mimics  the  act  of  the  helmsman 
observing  how  a  certain  rudder  change  is  affecting  the  course  heading  and  adjusting 
accordingly.  We  conducted  testing  to  choose  the  final  values  for  the  controller  inputs. 
The  end  result  was  the  design  of  a  highly  responsive  auto-helmsman  that,  when  given  a 
course  and  maximum  rudder  angle,  would  quickly  and  consistently  bring  the  ship  to  the 
ordered  heading. 

k.  Controller  HUD 

One  of  the  design  goals  was  to  devise  an  easy-to-use  interface  that  would  assist  in 
ship  driving.  The  focus  of  the  operator  was  to  be  on  the  task  at  hand,  rather  than 
interacting  with  an  overly  complex  control  interface.  The  image  in  Figure  13  shows  the 
overall  layout  of  the  interface  used  to  control  the  ship  and  presents  the  reports  that  would 
be  relayed  verbally  from  the  navigation  evaluator  to  the  conning  officer.  This  section 
outlines  the  purpose  of  each  segment  of  this  interface. 
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Figure  13.  Controller  Interfaee  as  Seen  by  the  Operator  (the  Experimenter) 


(1)  Ship  Controls 

The  image  in  Figure  14  represents  the  primary  interfaee  for  driving  the  ship. 


Ordered  Engine  Power:  0 

Ordered  Rudder;  0.00 

•■0.g 

H:  270 

10  (Xe 

fa: 

Figure  14.  Ship  Controls 


•  The  “Ordered  Engine  Power:”  this  eontrol  feeds  direetly  to  the  physies 
seripts  that  drive  the  ship.  The  baseline  value  of  five  was  programed  so 
that  under  ideal  eonditions  (no  set  or  drift  present)  the  ship  would  make 
preeisely  15  knots  through  the  water.  Type:  (input  eontrol) 
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•  The  “Ordered  Rudder:”  this  control  allows  for  manual  control  of  the  ship 
in  occasions  where  the  auto-helmsman  system  is  less  appropriate.  An 
example  would  be  when  a  conning  officer  orders  a  standard  rudder 
command  with  no  final  course  given.  Type:  (input  control) 

•  The  label  titled  “H:  270”  is  a  direct  value  of  the  ships  actual  heading  (not 
the  course  over  ground).  This  was  used  to  report  passing  headings  and 
keep  track  of  the  auto-helmsman.  Type:  (system  report) 

•  The  dark  text  input  box  with  “000”  is  a  way  to  input  the  ordered  course. 
Once  the  “AUTO  STEER”  checkbox  is  checked,  the  auto-helmsman 
system  takes  over  and  turns  the  ship  in  the  shortest  direction  toward  that 
course.  This  is  where  the  PID  controller  takes  over  and  attempts  to  zero-in 
on  the  course  requested.  The  “AUTO  EIEE”  checkbox  allows  the  system 
to  drive  itself,  receiving  input  on  the  next  course  from  the  waypoint 
system.  This  was  used  for  testing  and  demonstration  purposes.  Type: 
(input  control) 

•  The  Green  blocks  (“5  Deg,”  “10  Deg,”  “Standard”=15,  “EuU”=30, 
“Hard”=35)  are  used  as  governors  for  the  autopilot.  This  was  put  into 
place  because  conning  officers  can  specify  a  maximum  rudder  usage 
during  any  turn.  Type:  (input  control) 

(2)  Ship  Report 

The  image  in  Eigure  15  represents  a  display  mechanism  used  to  keep  track  of  the 
ship’s  location  relative  to  the  current  leg.  This  capability  was  mostly  used  during  the 
testing  phases,  and  it  was  kept  as  a  monitoring  tool  to  ensure  the  navigation  reports  were 
accurate.  The  values  presented  there  consisted  of:  (1)  ship  speed  (unit:  knots),  (2)  course 
over  ground — COG  (three  digit  number),  (3)  distance  to  turn — DTT  (unit:  yard),  and  (4) 
distance  off  track  (unit:  yard). 


Ship  Speed:  0.00  kts 
Course  Over  Ground:  000 
Distance  to  Turn:  644.3  yds. 
Distance  Off  Track:  0.0  yds. 


E igure  1 5 .  Ship  Report 
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(3)  Navigator  Report 

The  image  in  Figure  16  provides  the  details  of  a  dynamie  navigation  report  that 
summarizes  all  of  the  data  needed  for  the  aural  reports  and  formats  them  in  a  coneise 
script.  The  text  that  was  kept  constantly  updated  was  colored  either  yellow  or  purple.  The 
numbers  that  are  colored  purple  are  meant  to  be  read  out  individually  just  as  they  would 
be  on  the  ship,  e.g.,  330  is  read  “Three,  Three,  Zero.” 


At  this  time.  I  hold  the  ship 
center  of  track. 

Current  speed  is  0.0  knots. 
Course  over  ground  is 
Set  and  drift  is  negligible 
Distance  to  turn  is  645  yards 
Next  course  is 


Figure  16.  Navigator  Report 


While  the  image  in  Figure  16  shows  only  one  moment  in  time,  the  actual  reports 
were  dynamic  and  they  had  several  different  basic  forms. 

•  “1  hold  the  ship  N1  yards  (left/right)  out  of  the  turn.  Current  speed  is  N2 
kts.  Current  course  over  ground  is  XXX.  Distance  to  turn  is  N3  yds.  Next 
course  is  YYY.  At  this  time,  recommend  (come  (Right/Left)  to  regain 
track  /  maintain  course  and  speed).”  — This  form  of  the  report  was  shown 
at  the  moment  when  the  ship  was  determined  to  be  out  of  the  turn. 

•  “At  this  time,  1  hold  the  ship  N1  yards  (left  of/right  of/on)  track.  Current 
speed  is  N2  kts.  Current  course  over  ground  is  XXX.  Distance  to  turn  is 
N3  yds.  Next  course  is  YYY.  At  this  time,  recommend  (come  (Right/Left) 
to  regain  track  /  maintain  course  and  speed).”  — This  report  was  shown 
every  minute  (500  yds)  after  a  turn. 
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•  “200  yards  till  turn”  — This  form  of  the  report  was  shown  when  ship  was 
positioned  200  yards  before  the  turn. 

•  “100  yards  till  turn”  — This  form  of  the  report  was  shown  when  ship  was 
positioned  100  yards  before  the  turn 

•  “Recommend  come  to  course  XXX”  — This  form  of  the  report  was  shown 
when  ship  was  supposed  to  start  turning. 

(4)  Character  Controls 

The  image  in  Figure  17  shows  the  interactive  buttons  used  by  the  operator  to 
move  the  conning  officer’s  viewpoint  within  the  ship.  When  one  of  the  buttons  is 
selected,  the  system  smoothly  “walks”  the  viewpoint  to  the  desired  position  no  matter 
where  the  conning  officer  is  currently  positioned  on  the  bridge. 


,  J3haracter  Controls 

Left  Wing 

Right  Wing 

Left  Inside 

Right  Inskle 

Figure  17.  Character  Controls 


(5)  Network  and  Experiment  Setup 

The  primary  purpose  of  the  controller  displayed  in  Figure  18  was  to  establish  the 
network  connection  between  the  host  and  the  client.  The  secondary  purpose  was  to  create 
a  text  file  and  set  the  filename  by  concatenating  the  subject  identifier  and  system 
condition  that  was  ran  at  the  time.  The  text  input  box  labeled  “SUBJECT  ID”  is  where 
the  subject’s  identifier  code  was  typed  in  and  the  selection  buttons  underneath 
corresponded  to  the  three  conditions  being  tested  in  the  experiment.  The  box  underneath 
allowed  the  operator  to  type  in  the  Host  IP  address.  The  field  was  prepopulated  with  the 
address  of  the  computer  running  the  executable  code  and  only  the  last  triple  needed  to  be 
changed  (the  host  was  configured  to  be  on  the  same  router). 
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Figure  18.  Connection  Data 
(6)  Track  Status  Icon 

The  image  in  Figure  19  shows  the  tool  used  to  assess  the  ship’s  progression  on  the 
current  leg’s  track.  The  ship  icon  is  shifted  left  or  right  of  the  yellow  centerline,  to  mirror 
the  actual  ship  as  it  moved  along  the  track.  The  “N;  330”  represents  the  course  for  the 
next  leg  and  the  “C:  270”  is  an  indication  of  what  the  baseline  course  is  for  the  current 
leg.  A  numerical  value  appears  either  right  or  left  of  the  ship  icon  indicating  the  fact  that 
the  ship  is  left  or  right  of  the  track. 
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Figure  19.  Track  Status 
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(7)  Emergency  Buttons 

The  image  in  Figure  20  shows  two  interactive  buttons  that  we  designed  to  be  used 
only  in  rare  cases  or  in  testing.  The  “END  EXPERIMENT”  button  does  not  stop  the 
simulation,  but  rather  writes  all  the  data  generated  until  that  point  out  to  the  disk.  Due  to 
the  slow  read/write  times  on  the  disk,  all  data  is  held  in  memory  until  the  experimental 
session  was  over.  The  button  was  only  meant  to  be  used  if  the  subject  had  to  quit 
prematurely  or  if  something  went  wrong  with  the  experiment.  We  only  had  to  use  this 
button  once  in  the  entirety  of  the  user  study — in  this  instance,  we  had  the  subject  repeat 
that  transit. 

We  implemented  the  “SHIFT  WAYPOINT”  button  after  the  test  of  the  final 
system  revealed  a  flaw  in  which  the  waypoints  were  not  shifting  automatically.  This  rare 
case  only  occurred  when  the  subject  was  significantly  off  track  at  the  end  of  a  leg  (greater 
than  150  yards).  Actual  experiment  trials  required  the  use  of  this  button  only  three  times 
out  of  75  sessions. 


END  EXPERIMENT  SHIFT  WAYPOlffT 


Figure  20.  Emergency  Buttons 

1.  Conning  Officer  HUD 

The  primary  purpose  of  the  user  study  was  to  test  an  AR  conning  solution.  To 
support  that  goal  most  effectively,  designing  and  building  the  optimal  solution  of  the 
HUD  was  an  important  step  in  the  overall  engineering  process.  The  image  in  Figure  21 
shows  a  screen  capture  of  the  subject’s  view  with  the  HUD  deployed.  As  shown,  the 
HUD  consists  of  a  series  of  semi-transparent  indicators  that  present  a  concise  navigation 
picture  to  the  conning  officer.  The  image  in  Figure  21  shows  that  even  when  looking  at  a 
bright  sky,  the  numbers  and  letters  have  a  good  contrast  against  the  light  background  and 
they  are  still  perfectly  legible. 
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Figure  2 1 .  Conning  Stereo  View  with  HUD  Deployed  (Bridgewing) 


The  image  in  Figure  22  shows  a  screen  capture  of  the  HUD  from  the  monitor  that 
mirrored  the  images  displayed  in  the  HMD.  The  images  of  letters  appear  blurry  due  to  the 
stereoscopic  rendering  that  occurs  during  the  runtime.  As  with  the  control  GUI,  the 
standard  reports  of  COG  (course  over  ground),  SOG  (speed  over  ground),  and  DTT 
(distance  to  turn)  are  clearly  displayed.  The  “C:”  represents  the  current  leg’s  course  and 
the  “N:”  represents  the  next  leg’s  course.  The  yellow  vertical  line  is  a  representation  of 
the  track  which  stays  stationary  relative  to  the  HUD.  The  ship  icon  replicates  the  ship’s 
own  position  relative  to  the  track.  In  Figure  22  the  ship  is  17  yards  right  of  track  and  thus 
a  red  numerical  display  of  17  is  shown  just  right  of  the  ship  icon.  In  Figure  23,  the  ship  is 
98  yards  left  of  track  and  thus  a  green  numerical  display  of  98  is  shown  just  left  of  the 
ship  icon.  The  black  circle  with  blue  numbering  inside  is  an  indicator  of  relative  set  and 
drift  (Figure  22).  The  arrow  coming  off  of  it  points  in  the  direction  to  which  wind  and  sea 
(ocean)  currents  are  pushing  the  ship.  The  blue  text  inside  is  a  numerical  report  of  how 
much  force  is  being  exerted  against  the  ship  as  a  result  of  currents  (unit:  knots). 
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Figure  22.  HUD  Displaying  Set  and  Drift 


While  all  the  data  sets  presented  in  the  HUD  are  elear  and  easily  distinguishable 
from  eaeh  other  (3D  environment  and  text  overlay),  having  the  HUD  constantly  active 
during  the  transit  would  potentially  lead  to  cognitive  tunneling.  Cognitive  tunneling  is 
when  the  human  operator  is  focused  on  one  layer  of  information  while  completely 
disregarding  the  content  that  belongs  to  another  layer(s)  of  information  that  appears  in  the 
same  space.  In  a  real-world  scenario  where  shipping  traffic  and  navigation  hazards  are  a 
constant  distraction,  a  signal  for  an  upcoming  turn  might  not  be  immediately  recognized. 
Therefore,  we  attempted  to  mitigate  this  possibility  by  actively  alerting  the  subjects  that  a 
turn  was  approaching.  The  images  in  Figure  23  and  Figure  24  show  large  arrows  in  the 
center  of  the  HUD.  These  arrows  flash  on  and  off  in  one  second  increments  making  it 
more  likely  that  the  user  will  notice  them.  The  yellow  arrow  begins  flashing  200  yards 
before  the  turn  and  the  green  arrow  flashes  at  the  time  of  the  turn.  Whereas  real-world 
notification  for  an  upcoming  turn  occurs  at  around  1,000  yards,  the  compressed 
dimensions  of  the  environment  used  in  the  study  required  shifting  that  to  200  yards. 
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Figure  23.  HUD  Displaying  Upcoming  Turn  Notification 


COG : 

C: 

SOG : 

DTT: 

1 

1 

N: 

17yds 

Figure  24.  HUD  Displaying  Immediate  Turn  Notification 


We  discovered  a  critial  problem  during  development.  It  was  extremely  difficult 
for  users  to  read  the  text  when  looking  toward  backgrounds  with  lighter  colors.  The 
relatively  low  resolution  of  the  headset  combined  with  the  outdoor  lighting  of  the 
scenario  made  the  text  illegible  -  a  lack  of  contrast  made  the  text  very  hard  to  read.  A 
solution  to  this  problem  is  to  use  an  outlined  text  and  create  sufficient  amount  of 
constrasts  with  any  background.  Unfortunately,  neither  the  Unity  game  engine,  nor  the 
Oculus  Rift  scripts  provided  native  capability  to  display  outlined  text.  Therefore,  we 
instead  created  a  custom-made  solution  to  remedy  this  problem.  The  concept,  outlined  in 
the  image  in  Figure  25  involves  writing  each  letter  five  times.  The  first  four  times,  the 
lettering  is  offset  by  one  or  two  pixels  in  each  of  four  comer  directions.  The  fifth  print  is 
colored  in  the  desired  font  color  and  placed  directly  in  the  center  of  the  letter.  Because 
this  procces  would  be  required  for  every  single  character  written  to  the  HUD,  we 
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minimized  the  use  of  this  system  by  employing  it  only  for  the  elements  that  would 
ehange  during  the  session  (dynamie  text  elements  only).  The  image  in  Figure  26  shows 
the  file  that  was  used  as  the  background  of  the  HUD  to  eliminate  most  of  these  calls. 
Rendering  the  single  background  file  would  take  only  one  function  call,  whereas  the 
outlined  text  rendering  method  requires  several  calls;  reducing  the  number  of  characters 
that  needed  to  be  rendered  this  way  reduced  the  total  proccesing  time  in  the  final 
simulation. 


Step  1  Step  2  Step  3  Step  4  Step  5 

A  A  A  A  A 

^A  A 

Figure  25 .  Outlined  T ext  Rendering 


Figure  26.  HUD  Background  Image 


m.  Training  Scenario 

Due  the  novelty  of  fully  immersive  VR  experiences  and  the  interactive  methods 
employed  in  this  study,  we  assumed  that  our  user  pool  had  little  to  no  experience  with 
VR  simulations.  Given  this  assumption,  it  was  necessary  to  develop  a  training 
environment  that  would  be  used  to  teach  the  subjects  how  to  navigate  (move  about)  and 
perform  different  tasks  within  a  VR  simulation.  As  seen  in  the  image  of  Figure  27,  this 
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training  environment  represented  a  large  virtual  space  with  3D  objects  that  the  subject 
observed  and  walked  around  in.  In  the  scenario,  the  subject  was  asked  to  complete  three 
simple  tasks  that  had  all  major  characteristics  of  the  interaction  modalities  incorporated 
into  the  main  study. 


% 


Figure  27.  VR  Training  Simulation  Overview 
(1)  Instructions 

One  task  that  we  needed  to  teach  each  subject  was  to  deploy  (activate)  the  HUD. 
In  order  to  impart  this  skill  on  the  participants,  we  employed  the  action  of  tapping  the 
mask  at  least  four  times  within  the  training  simulation.  Tapping  the  mask  resulted  in  the 
HUD  textual  overlay  appearing  in  the  visual  field  of  view  inside  the  HMD.  Every  time 
the  system  required  the  user  to  take  action,  the  message  shown  in  the  image  of  Figure  28 
was  flashed  inside  the  HUD.  This  forced  the  subjects  to  actively  practice  tapping  the  side 
of  the  mask  with  their  fingers  while  simultaneously  teaching  them  how  much  force  they 
needed  to  use  for  successful  deployment  of  HUD  overlay. 
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Figure  28.  Instruction  Deployment  Method 


(2)  Navigation 

When  the  simulation  first  initializes,  the  subject  is  placed  in  the  center  of  a  large 
field.  The  environment  comprises  of  three  equally-sized  zones,  where  each  zone  is 
reached  by  using  a  rail-navigation  system.  The  user  is  asked  to  look  around  his  or  her 
environment  and  choose  which  colored  zone  they  would  like  walk  toward.  Upon  verbal 
request,  the  program  operator  then  presses  a  keyboard  key  and  the  user  is  walked  toward 
the  declared  zone. 

(3)  Yellow  Zone 

The  yellow  zone  consists  of  a  grey  brick  wall  with  four  pictures  hanging  on  it 
(Figure  29).  The  task  is  to  count  the  number  of  pictures  with  cats  on  them.  The  purpose 
of  this  task  is  to  force  the  users  to  look  around  within  the  environment  and  mentally 
process  the  scene.  Although  the  task  itself  is  very  simple,  it  forces  the  subject  to  rotate  his 
or  her  head  and  actively  look  around  while  scanning  the  entirety  of  the  environment. 
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Figure  29.  VR  Training  Simulation  (Yellow  Zone) 


(4)  Blue  Zone 

The  blue  zone  consists  of  two  cats  placed  approximately  20  and  30  feet  away 
from  the  subject  respectively  (Figure  30).  The  user  then  begins  the  task  of  determining 
which  of  the  felines  is  closer  to  them.  The  purpose  of  this  is  to  determine  if  the  subject  is 
having  difficulty  with  depth  perception  in  the  VE.  As  outlined  in  Chapter  II,  perceiving 
depth  in  a  VE  can  be  a  challenge.  Ship  driving  requires  the  ability  to  judge  distances; 
therefore  it  is  important  to  know  if  the  subject  has  difficulty  with  this  type  of  task  prior  to 
starting  the  primary  simulation. 


70 


m 


Figure  30.  VR  Training  Simulation  (Blue  Zone) 

(5)  Red  Zone 

The  red  zone  consists  of  one  cat  placed  approximately  10  feet  in  front  of  the 
subject  (Figure  31).  The  task  is  to  use  the  digital  compass  to  ascertain  the  bearing  of  the 
cat;  a  red  colored  number  displayed  in  the  middle  of  the  screen,  suggests  the  bearing  of 
the  object  in  the  center  of  screen,  and  it  gets  updated  dynamically  as  the  subject  moves 
his  or  her  head.  By  performing  this  task,  the  subject  learns  how  to  use  the  digital 
compass.  During  the  primary  simulation  this  tool  is  available  with  or  without  the  HUD 
elements,  so  it  was  important  for  the  test  subjects  to  familiarize  themselves  with  its 
presence  and  purpose. 
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please  use  direction  indicator  above  to 
aseartagi  tbe  direction  of  the  cat  nfron:  of  you 

3ifno»y  teil  the  operator  thejbirector’  r  vrrncf  ttw 
cat  i<5  located 


Figure  3 1 .  VR  Training  Simulation  (Red  Zone) 

C.  SUMMARY 

The  purpose  of  this  chapter  was  to  present  the  process  of  creating  the  environment 
and  the  different  solutions  needed  for  the  study,  including  the  solutions  that  had  to  be 
abandoned,  and  act  as  a  guide  for  those  who  might  go  through  creation  of  a  similar 
system.  The  technical  highlights  of  the  simulation  were  the  use  of  a  PID  controller  for  the 
helmsman,  the  simplistic  programming  approach  to  outlining  text  when  no  functions  are 
natively  available,  and  the  seamless  integration  of  a  brand  new  HMD  with  few  code 
repositories.  There  were  many  points  along  the  way  where  a  simpler,  good-enough,  and 
faster-to-code  solution  was  chosen  over  a  solution  that  would  have  been  more  true  to  the 
real-world  physics.  The  purpose  of  the  study  was  to  test  a  very  specific  idea  and  not  to 
invest  a  disproportional  amount  of  time  to  build  an  accredited  ship  driving  simulator.  We 
ensured  that  the  display  frame  rate  remained  around  75  FPS — this  was  done  to  avoid  a 
possibility  of  symptoms  of  cybersickness  caused  by  low  frame  rate.  This  goal  was 
achieved  by  limiting  the  number  of  in-game  objects  in  3D  environment.  The  officers  and 
engineers  who  either  tested  our  system  or  took  part  in  the  study  had  overwhelmingly 
positive  reviews  of  the  level  of  realism  presented  in  the  environment  which  meant  that 
the  variety  and  the  number  of  objects  in  the  scene  were  satisfactory;  this  type  of  feedback 

justified  the  design  decisions  that  had  to  be  made  along  the  way. 
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V.  USABILITY  STUDY  AND  DATA  ANALYSIS 


This  chapter  details  the  user  study  and  aceompanying  results.  Sections  in  this 
chapter  review  the  subjeet  pool,  the  experimental  methodology,  and  the  scenario  used  in 
the  study.  The  data  sets  that  were  analyzed  include  empirieal  measurements  (objective 
data  set),  subjects’  questionnaires,  and  interviews.  The  software  tools  utilized  for  the 
statistical  analysis  of  collected  data  were  JMP  Pro  Version  1 1  and  Statistiea  Version  10. 

A,  SUBJECTS 

The  study  engaged  highly  speeialized  subjects  with  a  very  specific  skill  set.  While 
the  U.S.  Navy  does  not  require  any  actual  certification  for  standing  the  watch  of  conning 
officer,  the  skill  set  required  to  carry  out  the  role  is  not  one  that  can  be  mastered 
overnight.  Early  on  in  the  development  phase  of  this  user  study,  we  established  that 
testing  would  oceur  in  institutions  with  signifieant  population  of  surfaee  warfare  offieers. 
By  reeruiting  subjects  in  these  plaees,  we  were  able  to  aequire  enough  qualified 
candidates  to  make  the  data  we  gathered  statistically  relevant.  Testing  oceurred  in 
Deeember  of  2014  at  Surfaee  Warfare  Officer  School  (SWOS),  Newport,  Rhode  Island. 
A  second  round  of  testing  oceurred  at  NPS,  Monterey,  California  in  January  of  2015. 

Contaet  was  made  with  SWOS  leadership  early  in  the  development  phase;  it  was 
agreed  that  we  would  not  only  utilize  their  expertise,  but  would  also  seek  eonsent  to 
conduct  experimentation  at  the  schoolhouse.  As  SWOS  is  the  U.S.  Navy’s  primary 
schoolhouse  for  the  training  of  ship  drivers,  it  is  also  a  very  logieal  place  to  test  new  ship 
driving  eoncepts.  With  the  approval  from  SWOS  leadership  to  reeruit  their  students  and 
instructors  for  our  study.  Institution  Review  Board  (IRB)  approval  was  promptly  applied 
for.  The  IRB  granted  approval,  and  we  began  to  advertise  for  the  study  through  email 
announeements,  personal  exchanges,  and  recruitment  flyers — the  text  of  the  flyer  is 
enclosed  in  Appendix  A.  We  traveled  to  Newport,  RI  for  one  week  to  conduct  the  study. 
By  the  end  of  the  week,  we  had  12  volunteers  for  the  study. 

We  performed  a  second  series  of  tests  at  NPS.  Similar  to  the  segment  of  the  study 
done  at  the  SWOS,  we  recruited  experienced  surface  warfare  officers  at  NPS.  By 
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advertising  to  the  NPS  surface  warfare  officer  group  via  email  and  on  the  NPS  daily 
muster  page  we  were  able  to  recruit  a  total  of  13  subjects,  and  complete  all  sessions 
within  one  week. 

We  asked  each  of  the  25  subjects  to  fill  out  a  personal  and  professional  history 
survey  at  the  beginning  of  the  study  (Appendix  B).  Although  the  study  was  advertised  to 
both  sexes,  all  of  our  subjects  were  male.  The  information  in  Table  2  shows  the 
distribution  of  the  age  amongst  the  subjects.  The  average  age  was  30.56  years  old  with  a 
standard  deviation  of  3.69  years.  The  youngest  subject  was  25  years  old  and  the  oldest 
was  39  years  old. 


Mean 

30.56 

Std  Dev 

3.6864617 

Std  Err  Mean 

0.7372923 

Upper  95%  Mean 

32.081697 

Lower  95%  Mean 

29.038303 

N 

25 

Table  2 .  Subj  ect  Age 


Eighty-eight  percent  of  the  subjects  were  professionally  designated  as  surface 
warfare  officers  (22  out  of  total  25).  The  remaining  three  subjects  (12%  of  our  sample) 
were  designated  as  information  professional  officer,  engineering  duty  officer,  and  deck 
surface  limited  duty  officer  respectively.  Ninety-six  percent  of  the  subjects  held  the  rank 
of  lieutenant  or  03.  One  subject  was  a  lieutenant  junior  grade;  in  this  case  the  subject  was 
prior  enlisted  and  thus  classified  as  an  02E.  The  information  in  Table  3  indicates  the 
distribution  of  years  of  active  duty  service  for  each  subject.  The  average  years  of  active 
duty  service  was  9.48  years  with  a  standard  deviation  of  5.42  years.  The  minimum  years 
of  active  duty  service  was  4  and  the  maximum  was  20. 
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Mean 

9.48 

Std  Dev 

5.4227146 

Std  Err  Mean 

1.0845429 

Upper  95%  Mean 

11.718387 

Lower  95%  Mean 

7.2416134 

N 

25 

Table  3.  Years  of  Aetive  Duty  Serviee 


The  information  in  Table  4  indicates  the  distribution  of  years  served  while 
designated  as  a  1160/1800,  the  U.S.  Navy  designations  for  qualified/unqualified  surface 
warfare  officers.  While  some  of  the  subjects  may  have  served  in  the  U.S.  Navy  for  many 
years,  this  statistic  gives  an  indication  of  how  much  time  they  would  have  spent  in  the 
surface  warfare  community.  The  average  number  of  years  as  a  1160/1800  designation 
was  5.22  years  with  a  standard  deviation  of  1.8  years.  The  minimum  years  reported  was 
zero  and  the  maximum  was  eight  years.  The  subject  that  reported  having  served  zero 
years  in  the  designation  had  been  prior  enlisted  for  over  a  decade  and  had  been 
previously  designated  as  a  boatswains  mate.  This  enlisted  rating  is  one  of  the  primary 
rates  responsible  for  driving  the  ship  and  thus  his  experience  level  was  equivalent,  if  not 
better,  than  the  other  subjects. 


Mean 

5.22 

Std  Dev 

1.8261982 

Std  Err  Mean 

0.3652396 

Upper  95%  Mean 

5.9738176 

Lower  95%  Mean 

4.4661824 

N 

25 

Table  4.  Years  Designated  as  1160/1800 
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The  average  time  since  each  subject  had  stood  the  watch  of  conning  officer  was 
18.76  months  with  a  standard  deviation  of  16.55  months.  The  minimum  time  since  last 
conning  was  zero  months  and  the  maximum  was  72  months.  The  average  time  since  each 
subject  had  last  stood  the  watch  of  conning  officer  during  a  restricted  maneuvering 
operation  was  21.96  months  with  a  standard  deviation  of  20.44  months.  The  minimum 
time  since  last  conning  during  a  restricted  operation  was  zero  months  and  the  maximum 
was  72  months.  The  information  in  Table  5  shows  the  distribution  of  the  number  of  times 
each  subject  estimated  they  had  stood  the  watch  of  either  OOD  or  conning  officer  while 
transiting  into  or  out  of  port.  The  average  number  of  transits  was  29.38  with  a  standard 
deviation  of  21.17  transits.  The  minimum  was  zero  transits  and  the  maximum  was  100. 
One  subject  did  not  choose  to  answer  this  question  (N  24  for  this  question). 


Table  5.  Times  OOD  /  Conn  out  of  port 


Out  of  the  25  subjects,  6  individuals  (24.00%)  had  been  qualified  as  a  navigator  or 
assistant  navigation  officer.  For  those  who  had  been  qualified,  the  average  time  serving  in 
that  position  was  1.64  years  with  a  standard  deviation  of  2.17  years.  Of  the  25  subjects, 
19  individuals  (76.00%)  had  served  onboard  an  ECDIS-N  certified  ship.  For  those  who 
had  done  so,  the  average  time  since  being  onboard  was  2.05  years  with  a  standard 
deviation  of  1.83  years.  When  asked  to  rate  their  current  skill  level  at  conning  ships,  the 
subjects  were  given  five  categorical  levels  to  choose  from.  These  were;  Bottom  5%,  35 
50%,  50  75%,  75  95%,  and  Top  5%.  Of  the  25  subjects,  6  individuals  (24.00%)  reported 
themselves  at  the  50%-75%  percentile,  12  individuals  (45%)  reported  themselves  at  the 
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75-95%  percentile,  and  seven  individuals  (28.00%)  reported  themselves  at  the  top 
5.00%. 


Of  the  25  subjects,  15  individuals  (60.00%)  reported  that  they  play  video  games. 
The  distribution  of  game  types  they  play  is  shown  in  Table  6.  The  most  commonly  played 
games  were  first-person  shooters  (56.00%)  and  adventure/fantasy/role-playing  (36.00%). 
The  most  common  platform  for  playing  video  games  was  a  game  console  followed  by  a 
computer. 
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GAME  TYPE  \  SYSTEM 

Com 

auter 

Smart 

phone 

Tablet,  load 

Game  Console 

Other  Cellphone 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

First  Person  Shooter 

7 

28.00% 

0 

0.00% 

0 

0.00% 

11 

44.00% 

0 

0.00% 

Online  Multiplayer 

5 

20.00% 

0 

0.00% 

0 

0.00% 

4 

16.00% 

0 

0.00% 

Adventure,  Fantasy, 
Role  Playing 

6 

24.00% 

0 

0.00% 

0 

0.00% 

5 

20.00% 

0 

0.00% 

Other: 

6 

24.00% 

2 

8.00% 

1 

4.00% 

3 

12.00% 

1 

4.00% 

Table  6.  Subject  Video  Game  Practice 
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The  survey  given  ineluded  a  section  about  previous  experience  with  simulators. 
Every  subject  reported  having  prior  experience  with  simulators  and  24  of  25  (96.00%) 
reported  having  used  COVE  at  some  point  in  their  career.  Eor  those  who  reported  having 
used  COVE,  the  information  in  Table  7  indicates  the  distribution  of  the  number  of  hours 
utilizing  the  trainer.  The  average  time  since  last  use  was  19.34  months  with  a  standard 
deviation  of  20.16  months.  The  second  highest  reported  simulator  was  the  FMB  with  10 
of  25  subjects  (40.00%)  reporting  that  they  had  used  the  system.  The  average  time  spent 
in  the  FMB  was  44.90  hours  with  a  standard  deviation  of  63.06  hours  and  the  time  since 
last  use  average  was  25.89  months  with  a  standard  deviation  of  17.67  months.  Other 
reported  simulators  included  an  ECS  trainer,  a  point  defense  trainer  for  small  boat 
operations,  and  a  VMS  trainer. 


350 


300 


Mean 

71.541667 

Std  Dev 

88.341225 

Std  Err  Mean 

18.032577 

Upper  95%  Mean 

108.84489 

Lower  95%  Mean 

34.238439 

N 

24| 

Table  7.  Hours  Using  COVE 


Of  the  25  subjects  surveyed,  24  individuals  (96.00%)  reported  having  used  an 
HMD  before  and  only  one  (4.00%)  reported  having  used  an  AR  display. 

B.  STUDY  DESIGN 

The  goal  of  the  study  was  to  determine  the  viability  of  an  operational  setup  of  the 
bridge  onboard  U.S.  Navy  warships,  in  which  a  conning  officer’s  visual  field  is 
augmented  with  an  overlay  of  CNI  that  is  typically  relayed  in  aural  form  from  the 
navigation  officer.  This  work  specifically  targeted  the  bridge  experience  during  a 
restricted  navigation  transit. 
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1,  Physical  Environment 

For  both  the  SWOS  and  the  NPS  testing  cyeles,  we  kept  the  physieal  setup  of 
eomputers,  participants,  and  desks  the  same.  The  subject  fdled  out  paperwork  at  a  nearby 
desk  in  the  room.  The  subject  then  did  all  VR  sessions  while  standing  in  front  of  a  large 
desk  with  two  computers  and  two  monitors  atop  it.  As  shown  in  the  image  in  Figure  32, 
the  simulated  helmsman  and  navigation  evaluator  stood  behind  the  desk  and  the  subject 
(conning  officer)  stood  in  front  of  it.  We  left  enough  of  a  gap  in  between  the  monitors  to 
physically  observe  the  subject  for  outward  signs  of  cybersickness.  The  desk  also  gave  the 
subjects  something  to  hold  onto  in  the  case  of  disorientation  during  the  sessions.  The 
images  in  Figure  33  and  Figure  34  show  the  physical  layouts  of  personnel  and  equipment 
at  both  SWOS  and  NPS  respectively.  The  image  in  Figure  35.  shows  all  the  peripheral 
equipment  used  in  the  study. 


LEGEND: 


! 

! 
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Helmsman 
Navigation  Evaluator 
Conning  Officer 
Host  Computer 
HMD  Replication 
Client  Computer 
[W]  Controller  Interface 


Figure  32.  Physical  Study  Environment  Layout 
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Figure  33.  Physical  Study  Environment  (SWOS) 


Figure  34.  Physical  Study  Environment  (NPS) 
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Figure  35.  Physical  Study  Equipment 


2,  Support  Staff  (Experimenters) 

During  the  study,  two  researchers  played  the  parts  of  a  helmsman  and  a 
navigation  evaluator.  A  trained  officer  with  intimate  knowledge  and  experience  of  real- 
world  ship  driving  acted  as  the  helmsman.  This  officer  accurately  portrayed  a  helmsman 
through  accurate  repeat-backs  and  quick  actions  in  manipulating  the  ship.  The  part  of  the 
navigation  evaluator  was  played  by  a  team  member  who  had  been  trained  to  read  the 
automatically  updated  scripts  (detailed  in  Chapter  fV)  and  also  provide  additional  reports 
as  needed  based  on  the  information  available.  At  both  SWOS  and  NPS,  the  exact  same 
two  individuals  performed  these  tasks  for  all  sessions  for  all  subjects,  which  provided  a 
consistent  input  and  conditions  for  all  subjects. 

3,  Experimental  Conditions 

In  this  experiment,  subjects  conned  ships  using  the  NAVFIUD  ship  driving 
simulation  described  in  Chapter  IV.  In  order  to  test  the  hypotheses  outlined  in  Chapter  I, 
each  subject  conducted  three  transits  out  of  the  same  channel  in  each  condition  (within- 
subjects  study  design).  We  counterbalanced  experimental  conditions  to  avoid  learning- 
effects  influencing  results.  There  were  three  experimental  conditions  tested  in  user  study: 
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A:  Navigation  Evaluator  (CNI  provided  in  auditory  form  only):  The  subjeet 
reeeived  only  auditory  reports  by  a  trained  experimenter  who  played  a  role  of  a 
navigation  officer. 

B:  Navigation  Evaluator  +  HEID  (CNI  provided  in  both  auditory  and  visual  form): 
The  subjects  received  auditory  reports  in  addition  to  visual  information  (overlaid  in  their 
visual  field)  presented  inside  a  heads-up  display.  This  augmented  layer  is  the  conning 
officer  HUD  detailed  in  Chapter  IV. 

C:  HUD  (CNI  provided  in  visual  form  only):  The  subject  received  only  the  HUD 
elements,  i.e.,  visual  information  as  detailed  in  Chapter  IV. 

4.  Description  of  Ship  Navigation  Course 

Described  in  Chapter  IV,  we  specifically  built  the  fictitious  channel  created 
during  the  design  phase  to  provide  useful  data  for  the  study.  Table  8  provides  the  details 
about  each  leg  within  the  transit  out  of  the  channel.  Eigure  36  pictorially  represents  the 
same  data  but  overlays  each  leg  onto  a  chart.  The  ship  started  each  session  on  Eeg  0  at  a 
standstill  pointing  in  the  correct  direction  of  transit.  When  the  subject  was  ready  to  begin, 
the  experimenter  acting  as  helmsman,  brought  the  ship  up  to  speed.  Eor  the  purposes  of 
data  review,  we  collected  Eeg  0  data  but  did  not  consider  it  in  the  final  data  analysis.  This 
is  because  the  ship  started  on  track  and  the  conning  officer  needed  to  take  no  action  to 
keep  it  that  way.  Data  from  Eeg  1  was  the  first  that  we  used.  To  give  the  conning  officers 
a  slight  acclimation  period  to  how  the  ship  handled,  Eeg  1  did  not  include  any  current  to 
drive  the  ship  off  course.  The  rest  of  the  track  had  current  that  changed  in  direction  and 
speed  for  each  new  leg.  Each  leg,  with  the  exception  of  the  starting  leg,  was  1,500  yards 
long. 
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Leg 

Course 

Distance 

Direction  of 
Currents 

Speed  of  Currents 

0 

270 

1000  yds 

- 

0.0  kts 

1 

330 

1500  yds 

- 

0.0  kts 

2 

025 

1500  yds 

125 

1.2  kts 

3 

300 

1500  yds 

230 

2.5  kts 

4 

225 

1500  yds 

135 

2.0  kts 

5 

270 

1500  yds 

000 

1.7  kts 

Table  8.  Outbound  Transit  Leg  Details 
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Figure  36.  Outbound  Transit  Chart  with  Legs  and  Currents 
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5,  Bridge  Positions 

As  described  in  Chapter  IV,  the  study  allowed  the  subjects  to  move  to  several 
positions  on  the  bridge.  The  positions  in  the  study  included  two  “centerline”  positions 
(middle  of  the  bridge)  and  the  left/right  bridge  wings  positions  respectively.  The  conning 
officers  were  required  to  verbally  ask  the  helmsman  to  move  between  the  four  possible 
positions  on  the  bridge;  the  left  bridge  wing,  the  right  bridge  wing,  the  left-center  bridge, 
and  the  right-center  bridge.  The  image  in  Figure  37  depicts  a  view  from  the  left  bridge 
wing  (Position  A),  the  image  in  Figure  38  depicts  a  view  from  the  left-center  bridge 
(Position  B),  the  image  in  Figure  39  depicts  a  view  from  the  right-center  bridge  (Position 
C),  and  the  image  in  Figure  40  depicts  a  view  from  the  right  bridge  wing  (Position  D). 


Figure  37.  Stereo  View  from  Left  Bridge  Wing  Position 


Figure  38.  Stereo  View  from  Left-Center  Bridge  Position 
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Figure  39.  Stereo  View  from  Right-Center  Bridge  Position 


Figure  40.  Stereo  View  from  Right  Bridge  Wing  Position 

6,  Collection  of  Objective  Data  Set 

The  simulation  reeorded  data  for  eaeh  subjeet  in  three  separate  files,  eaeh 
eorresponding  to  one  experimental  eondition.  Data  reeorded  ineluded  the  following 
elements:  (a)  performanee  data  (distanee  from  the  traek),  (b)  instanees  when  the  HUD 
layer  was  deployed,  i.e.,  turned  ON  or  OFF,  and  (e)  subjeet’s  position  on  the  ship’s 
bridge  (ineluding  instanees  when  that  position  was  ehanged).  The  simulation  reeorded 
snapshots  of  data  at  even  distance  intervals  for  each  subject.  During  each  leg,  the 
simulation  constantly  calculated  DTT,  which  determined  when  to  take  data 
measurements.  When  the  ship  began  each  leg,  the  initial  value  of  DTT  would  be  slightly 
less  than  1,500  yards  due  to  advance  and  transfer  calculations.  As  the  ship  approached  the 
upcoming  turn,  that  DTT  would  approach  zero  and  then  reset  to  approximately  1,500 
yards  again  once  the  turn  started.  Every  ten  yards,  the  simulation  recorded  the  current 
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position  of  the  ship.  In  the  real  world,  as  the  ship  turns,  there  is  no  way  to  instantly 
determine  whether  it  is  left  or  right  of  traek;  this  led  us  to  deeide  not  to  begin  taking 
measurements  until  the  DTT  reaehed  1,350  yards,  i.e.,  when  we  were  eonfident  that  the 
ship  had  eompleted  the  turn  and  was  on  a  steady  eourse. 

7.  Collection  of  Subjective  Data  Set 

Throughout  the  study,  we  eolleeted  subjeetive  data  from  eaeh  subjeet.  These 
ineluded:  (1)  standard  simulator  siekness  questionnaire  (SSQ)  (Appendix  D)  at  the 
beginning  of  the  study  (to  form  a  baseline  SSQ  data  set)  and  after  eaeh  instanee  in  whieh 
they  donned  the  HMD,  (2)  opinion  questionnaires  were  given  after  eaeh  session  with  the 
HMD  (Appendix  E),  and  (3)  an  exit  interview.  The  goal  of  eolleeting  a  subjeetive  data  set 
was  to  supplement  the  objeetive  data  with  information  that  explained  or  elarified  the  final 
results.  Knowing  that  the  performanee  gains  were  registered  in  one  experimental 
eondition  over  the  others,  for  example,  would  not  explain  the  reason  why  that  happened. 
The  subjeetive  data  also  gave  us  additional  indieators  of  user  preferenees,  whieh  the  U.S. 
Navy  ean  use  in  deeiding  whether  an  option  warrants  adaption.  As  our  seeondary 
hypothesis  deals  with  the  end-users  aeeeptanee  and  integration  of  AR  into  daily 
operations,  we  needed  a  subjeetive  data  set  to  make  assertions  on  this. 

C.  METHODOLOGY 

We  eentered  the  study  on  immersing  the  subjeets  in  the  VR  environment 
deseribed  in  Chapter  IV  and  measuring  their  performanee.  Additionally,  throughout  the 
study,  we  eolleeted  eybersiekness  surveys,  questionnaire  data,  and  interview  data.  We 
used  the  eheeklist  in  Appendix  F  to  ensure  all  steps  were  taken  for  eaeh  subjeet  and  in 
exaetly  the  same  order.  The  following  list  presents  a  detailed  deseription  of  stages  that 
eaeh  subjeet  experieneed  in  the  study. 

1.  Pre-Experiment 

The  subjeet  filled  out  an  IRB  informed  eonsent  form  and  audiovisual  eonsent 
form  (Appendix  C).  They  then  filled  out  a  baekground  questionnaire  and  a  simulator- 
siekness  questionnaire  (Appendix  D).  The  questions  ineluded;  demographie  data. 
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information  about  past  experiences  within  the  study  domain,  past  experience  with 
computer-supported  simulations,  and  an  estimate  of  their  skill  levels  relevant  to  the  study. 
This  process  took  approximately  five  minutes. 

2,  Virtual  Environment  Training  Scenario 

The  subject  then  donned  a  head  mounted  display  (HMD);  once  the  device  was 
comfortably  fitted,  an  acclimation  simulation  with  a  training  scenario  was  started  (details 
provided  in  Chapter  IV).  This  simulation  enabled  the  subject  to  become  comfortable  with 
navigating  through,  and  interacting  with,  a  VE  using  the  same  interaction  modalities  that 
are  needed  in  the  main  study.  The  environment  was  a  representation  of  a  large  virtual 
space  with  three-dimensional  objects  that  the  subject  observed  and  walked  around  in.  In 
this  scenario,  the  subject  performed  simple  navigational  tasks  and  interaction  modalities 
that  consisted  of  major  characteristics  from  the  main  study.  The  subject  had  up  to  ten 
minutes  to  accomplish  three  tasks  in  the  training  environment;  after  which,  he  removed 
the  headset  and  answered  a  SSQ.  This  process  took  approximately  15  minutes.  25  of  25 
subjects  (100%)  had  no  difficulty  with  the  training  simulation  and  successfully  completed 
all  required  tasks. 

3.  Main  Experiment 

The  subject  read  the  information  about  the  task  he  would  be  completing  in  the 
upcoming  scenario  from  the  data  sheet  included  in  Appendix  G.  The  sheet  briefed  the 
subject  on  vessel  characteristics,  engine  orders,  rudder  orders,  internal  navigation 
procedures  on  the  bridge,  and  the  overall  transit  plan.  The  transit  plan  detailed  the 
expected  courses,  turns,  and  speeds  of  the  transit.  We  gave  each  subject  as  much  time  as 
necessary  to  study  the  transit  and  to  ask  questions  about  the  environment — no  subject 
spent  more  than  five  minutes  to  complete  this.  After  a  subject  reported  being  comfortable 
with  the  transit  plan  and  ship  specifications,  he  was  then  given  an  instruction  script  that 
depended  on  which  scenario  variation  was  presented  first.  This  script  outlined  the  task, 
the  situation,  and  the  mode  of  navigation  reports  that  were  provided  in  the  given 
experimental  condition  (Appendix  H).  On  starting  the  simulation,  the  subject  donned  the 
HMD  once  more. 


89 


Upon  donning  the  HMD,  the  subject  conned  the  vessel  along  the  track.  Each 
transit  took  approximately  15  minutes.  After  the  ship  had  crossed  the  finish  point,  the 
subject  removed  the  HMD  and  fdled  out  a  survey  that  captured  the  subject’s  opinions  on 
the  overall  experience  as  well  as  elements  of  SSQ,  which  took  approximately  five 
minutes. 

The  subject  repeated  this  entire  process  twice  to  account  for  each  of  the  three 
experimental  conditions  outlined  in  the  study  design.  Each  time  the  subject  was  briefed, 
conducted  a  transit  out  of  the  channel,  and  filled  out  a  survey  of  their  experience  within  a 
particular  scenario  as  well  as  a  SSQ.  The  entire  process,  including  the  accompanying 
paperwork,  took  1  hour  and  5  minutes  per  subject. 

4,  Post  Experiment 

The  subject  fdled  out  an  exit  survey  of  his  experiences  (Appendix  E).  In  addition, 
we  conducted  a  short  interview  to  gamer  opinions  that  were  not  captured  in  the  survey 
and  the  subject  was  given  a  debriefing  form  (Appendix  I). 

D,  RESULTS 

1.  Objective  Data  Set 

This  section  presents  a  summary  of  the  analysis  done  on  the  objective  data  set  that 
we  collected  during  the  study.  The  data  set  includes:  (1)  conning  officer  performance 
data,  (2)  conning  officer  bridge  position  analysis,  and  (3)  HUD  deployment  tracking 
analysis.  The  tables  in  Appendix  J  show  the  distribution  of  conning  officer  performance 
per  condition.  Under  Condition  A,  conning  officers  averaged  3 1 .90  yards  off  track.  Under 
condition  B,  conning  officers  averaged  17.41  yards  off  track.  Under  condition  C,  conning 
officers  averaged  14.80  yards  off  track.  In  order  to  determine  the  statistical  difference 
between  groups  we  needed  to  mn  an  analysis  of  variance  (ANOVA).  This  was 
determined  to  be  an  unsuitable  solution  due  to  the  data  not  having  normality.  This  lack  of 
normality  is  shown  in  distribution  graphs  in  Appendix  K.  In  order  to  compensate  for  the 
lack  of  normality  of  the  data,  we  ran  Mann- Whitney  U  Tests  between  each  of  the 
conditions.  The  greatest  significance  appears  between  Conditions  A  and  B  and 
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Conditions  A  and  C.  The  information  in  Table  9  and  Table  10  show  that  all  legs,  with  the 
exeeption  of  Leg  1,  have  p-values  less  than  .005.  Leg  1  has  less  signifieant  differenee  in 
performanee  due  to  the  laek  of  currents  on  that  leg.  The  relative  ease  of  that  leg  resulted 
in  all  conditions  reporting  similar  performances.  The  information  in  Table  11  shows  the 
analysis  between  Conditions  B  and  C.  The  p-values  in  this  case  average  above  .05 — 
indicating  a  lack  of  significance  between  the  conditions.  Since  both  Conditions  B  and  C 
utilized  the  HUD  and  Condition  A  did  not,  we  summarize  that  the  HUD  was  the 
significant  factor  in  an  increase  of  performance  per  session.  This  indicates  that  the 
reported  averages  are  statistically  significant  enough  to  reject  the  null  hypothesis 
presented  in  Chapter  1. 

•  HIq:  A  lightweight  glasses-type  AR  overlay  system  for  a  conning  officer 
does  not  increase  his  ability  to  maintain  a  close  proximity  to  a  preplanned 
track. 

Rejecting  Hlo  does  not  prove  our  alternate  hypothesis,  but  it  does  indicate  that  the 
HUD  had  a  positive  effect  on  the  conning  officer’s  overall  performance. 
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variable 

Mann-Whitney  U  Test  (NPS  Augme 
By  variable  Session 

Marked  tests  are  significant  at  p  <. 

nted  Reality  Conning  Officer  Performance  Study  CompressedData-averagesPerTrack-2015-02-23) 

D5000 

Rank  Sum 
A 

Rank  Sum 
B 

U 

Z 

p^ralue 

Z 

adjusted 

p^alue 

Valid  N 
A 

Valid  N 
B 

2*1sided 
exact  p 

Track  1 

707.0000 

568.0000 

243.0000 

1.338797 

0.180638 

1  338797 

0.180638 

25 

25 

0  182269 

Track  2 

784.0000 

491.0000 

166,0000 

2.832816 

0.004614 

2.832816 

0.004614 

25 

25 

0.003972 

Track  3 

880.0000 

395.0000 

70.0000 

4.695490 

0.000003 

4.695490 

0,000003 

25 

25 

0.000000 

Track  4 

799.0000 

476.0000 

151.0000 

3.123859 

0.001785 

3.123859 

0.001785 

25 

25 

0.001393 

Track  5 

802.0000 

473.0000 

148.0000 

3.182067 

0.001462 

3.182067 

0.001462 

25 

25 

0.001114 

Table  9.  Conning  Officer  Performance  (Condition  A  versus  Condition  B) 


variable 

Mann-Whitney  U  Test  (NPS  Augmented  Reality  Conning  Officer  Performance  Study  CompressedData-averagesPerTrack-2015-02-23) 
By  variable  Session 

Marked  tests  are  significant  at  p  <  05000 

Rank  Sum 
A 

Rank  Sum 
C 

U 

z 

p-value 

z 

adjusted 

p-value 

Valid  N 
A 

Valid  N 
C 

2*1  sided 
exact  p 

Track  1 

577.0000 

698.0000 

252,0000 

-1.16417 

0.244356 

-1.16417 

0.244356 

25 

25 

0.246748 

Track  2 

838.0000 

437.0000 

112.0000 

3.88057 

0.000104 

3,88057 

0.000104 

25 

25 

0  000050 

Track  3 

861.0000 

414.0000 

89.0000 

4.32684 

0.000015 

4.32684 

0.000015 

25 

25 

0.000004 

Track  4 

865.0000 

410.0000 

85.0000 

4.40445 

0.000011 

4.40445 

0.000011 

25 

25 

0.000003 

Track  5 

865.0000 

410.0000 

85.0000 

4.40445 

0.000011 

4.40445 

0.000011 

25 

25 

0.000003 

Table  10.  Conning  Officer  Performance  (Condition  A  versus  Condition  C) 


92 


variable 

Mann-Whitney  U  Test  (NPS  Augmented  Reality  Conning  Officer  Performance  Study  CompressedData-averagesPerTrack-2015-02-23) 
By  variable  Session 

Marked  tests  are  significant  at  p  <  05000 

Rank  Sum 
B 

Rank  Sum 
C 

U 

z 

p-value 

Z 

adjusted 

p-value 

Valid  N 
B 

Valid  N 
C 

2*1  sided 
exact  p 

Track  1 

520.0000 

755.0000 

195.0000 

-2.27013 

0.023200 

-2.27013 

0.023200 

25 

25 

0.022262 

Track  2 

716.0000 

559.0000 

234.0000 

1.51342 

0.130173 

1.51342 

0.130173 

25 

25 

0.131021 

Track  3 

683.0000 

592.0000 

267.0000 

0  87313 

0.382594 

0.87313 

0.382594 

25 

25 

0.385853 

Track  4 

750.0000 

525.0000 

200.0000 

2  17312 

0.029772 

2.17312 

0.029772 

25 

25 

0.028881 

Track  5 

751.0000 

524.0000 

199.0000 

2.19252 

0.028343 

2.19252 

0.028343 

25 

25 

0.027439 

Table  1 1 .  Conning  Officer  Performance  (Condition  B  versus  Condition  C) 
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The  charts  in  Figure  41,  Figure  42,  and  Figure  43  indicate  the  averages  of  how 
well  conning  officers  performed  for  each  leg — experimental  condition  being  the  variable 
factor  between  charts.  The  graphics  separate  the  performance  by  colors  where  dark  green 
indicates  the  subjects  were  less  than  10  yards  off  track  and  dark  red  indicates  they  were 
over  100  yards  off  track.  The  other  colors  are  clearly  labeled  in  the  legend.  Leg  0  data  is 
not  displayed  because  the  initial  starting  leg  placed  the  subjects  in  the  center  of  the  track 
and  required  no  external  manipulation  to  maintain  the  course — under  every  condition  the 
bar  would  be  pure  dark  green.  There  is  little  disparity  in  Leg  1  because  of  the  ease  of  that 
leg — due  to  a  lack  of  currents.  From  Leg  2  forward,  the  separation  of  performance 
becomes  clear.  In  all  session.  Leg  3  proved  to  be  the  most  difficult:  in  the  graphs  this  can 
be  seen  by  the  fact  that  the  lowest  percentage  of  dark  green  occurs  on  that  leg.  By 
reviewing  all  three  graphs,  a  clear  delineation  of  which  conditions  saw  better 
performance  from  the  subjects  can  be  seen.  Conditions  B  and  C  doubled  the  amount  of 
time  within  the  zero  to  ten  yards  zone  then  Condition  A.  Compiling  the  data  from  the 
information  in  Figure  41,  Figure  42,  and  Figure  43,  we  calculate  that  the  actual  time  on 
track  for  each  condition  as:  31.43%  (Condition  A),  47.10%  (Condition  B),  and  54.67% 
(Condition  C).  This  indicates  that,  when  given  the  option  of  having  the  data  presented  in 
the  HUD,  subjects  stayed  less  than  ten  yards  off  track  61.76%  more  of  the  time. 
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Percentage  of  Leg  Length  Percentage  of  Leg  Length 
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Figure  41 .  Subject  Performance  per  Leg:  Condition  A 
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Figure  42.  Subject  Performance  per  Leg:  Condition  B 
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Figure  43.  Subject  Performance  per  Leg:  Condition  C 


The  information  in  Table  12  shows  the  distribution  of  how  many  times  a  subjects 
activated  the  HUD  during  Condition  B.  The  results  indicate  that,  during  Condition  B,  the 
subject  activated  the  HUD  an  average  of  4.52  times  per  transit  with  a  standard  deviation 
of  5.29  times.  The  information  in  Table  13  shows  the  distribution  of  how  many  times  a 
subjects  activated  the  HUD  during  Condition  C.  The  results  indicate  that,  during 
Condition  C,  the  subject  activated  the  HUD  an  average  of  5.00  times  per  transit  with  a 
standard  deviation  of  10.07  times.  The  information  in  Table  14  shows  the  distribution  of 
how  many  times  a  subject  activated  the  HUD  over  both  Condition  B  and  Condition  C. 
The  results  indicate  that,  on  average,  the  subject  activated  the  HUD  4.76  times  per  transit 
with  a  standard  deviation  of  8.34  times.  One  subject  activated  the  HUD  51  times  and  19 
subjects  activated  the  HUD  only  once  during  a  session,  which  suggests  that  they  chose  to 
keep  it  deployed  for  the  entirety  of  the  transit. 
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Mean 

4.52 

Std  Dev 

5.2924474 

Std  Err  Mean 

1.0584895 

Upper  95%  Mean 

6.7046149 

Lower  95%  Mean 

2.3353851 

N 

25 

Table  12.  HUD  Activation:  Condition  B 


Mean 

5 

Std  Dev 

10.665365 

Std  Err  Mean 

2.1330729 

Upper  95%  Mean 

9.4024461 

Lower  95%  Mean 

0.5975539 

N 

25 

Table  13.  HUD  Activations:  Condition  C 
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Table  14.  HUD  Activations:  Condition  B  and  Condition  C 


Mean 

4.76 

Std  Dev 

8.3362009 

Std  Err  Mean 

1.1789168 

Upper  95%  Mean 

7.1291221 

Lower  95%  Mean 

2.3908779 

N 

50 
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The  images  in  Figure  44  and  Figure  45  display  the  pereentages  of  time  that 
subjects  kept  the  FIUD  deployed  in  both  Condition  B  and  C.  In  Condition  B,  subjects 
choose  to  keep  the  HUD  up  for  96%  of  the  transit.  In  Condition  C,  subjects  choose  to 
keep  the  HUD  deployed  for  98%  of  the  transit.  We  interpret  this  to  mean  the  subjects  felt 
very  comfortable  with  having  the  HUD  activated  for  the  entire  transit. 


Figure  44.  HUD  Deployment  Percentage:  Condition  B 


Figure  45.  HUD  Deployment  Percentage:  Condition  C 


The  information  in  Table  15  indicates  the  distribution  of  number  of  times  subjects 
chose  to  move  within  the  ship’s  bridge.  Of  the  75  total  sessions,  the  average  number  of 
transitions  was  3.65  times  per  session  with  a  standard  deviation  of  3.14  moves.  The  max 
number  of  moves  was  12,  and  9  subjects  chose  to  not  move  at  all. 
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14 


Mean 

3.6533333 

Std  Dev 

3.1429601 

Std  Err  Mean 

0.3629178 

Upper  95%  Mean 

4.3764628 

Lower  95%  Mean 

2.9302039 

N 

75 

Table  15.  Total  Moves  on  the  Bridge 


The  image  in  Figure  46  displays  the  aggregate  distribution  of  where  subjects 
chose  to  stand  in  the  simulation  through  all  sessions.  The  positions  are  labeled  as  follows: 
A  (the  left  bridge  wing),  B  (the  left-center  bridge),  C  (the  right-center  bridge),  and  D  (the 
right  bridge  wing).  Subjects  spent  the  most  time  (36%)  at  the  left-center  bridge  position. 
This  is  expected  as  that  position  was  the  starting  position  and  several  subjects  choose  not 
to  move  once  the  simulation  began.  The  second  position  with  the  highest  value  (27%)  is 
the  right-center  bridge.  This  indicates  that  subjects  preferred  to  be  inside  while  driving 
the  ship.  The  best  estimation  of  why  this  is  the  case  is  because  the  positions  closest  to 
centerline  provided  the  easiest  indicators  of  where  the  ships  bow  was  actually  pointing. 
The  image  in  Figure  47  displays  the  distribution  of  the  subjects’  location  per  session.  In 
each  session,  subjects  spent  the  most  time:  33%  (Condition  A),  36%  (Condition  B),  and 
38%  (Condition  C)  at  position  B.  At  each  position,  there  is  only  minor  statistical  variance 
per  condition  which  indicates  that  the  subjects’  position  on  the  bridge  was  not  determined 
by  their  use  of  the  HUD. 
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Figure  46.  Distribution  of  the  Subjeets’  Loeation  on  the  Bridge:  All  Conditions 
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Figure  47.  Distribution  of  the  Subjeets’  Loeation  on  the  Bridge:  Individual 

Conditions 


2,  Subjective  Data 

We  derived  the  data  set  presented  and  diseussed  in  this  seetion  from  the  surveys 
that  the  subjeets  took  after  eaeh  session  including  a  final  questionnaire  that  was  given 
after  the  final  session.  The  final  questionnaire  asked  the  subjects  to  order  the  conditions 
in  terms  of  usefulness  for  staying  close  to  the  centerline  of  track  (Appendix  E). 
Additionally,  the  final  questionnaire  asked  for  the  subjects’  opinions  on  the  realism  of  the 
VE. 

(1)  Post  Session  Surveys 

The  information  in  Table  16,  Table  17,  and  Table  18  present  what  subjects 
reported  when  asked  how  much  of  the  time  they  felt  they  were  on  track  (<10  yds  off 

track)  after  each  session.  Eor  Condition  A,  the  subjects  reported  they  felt  they  were  on 
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track  an  average  of  38.40%  of  the  total  transit  time  with  a  standard  deviation  of  19.08%. 
For  Condition  B,  the  subjects  reported  that  they  felt  they  were  on  traek  an  average  of 
61.60%  of  the  total  transit  time  (st.  dev.  22.30%).  For  Condition  C,  the  subjeets  reported 
they  felt  they  were  on  traek  an  average  of  64.00%  of  the  total  transit  time  (st.  dev. 
18.26%).  This  indieates  that  the  subjeets  partieipating  in  eonditions  with  the  HUD  felt 
they  were  on  traek  for  at  least  20%  more  of  the  time  than  without  the  HUD.  The  objeetive 
data  presented  in  the  previous  section  verifies  the  subjects’  opinions. 


Mean 

38.4 

Std  Dev 

19.078784 

Std  Err  Mean 

3.8157568 

Upper  95%  Mean 

46.275335 

Lower  95%  Mean 

30.524665, 

N 

25' 

Table  16.  Subjeet’s  Self-Estimation  of  Pereent  Time  On-Traek:  Condition  A 


Mean 

61.6 

5td  Dev 

22.300972 

Std  Err  Mean 

4.4601943 

Upper  95%  Mean 

70.805389 

Lower  95%  Mean 

52.394611 

N 

25 

Table  17.  Subject’s  Self-Estimation  of  Percent  Time  On-Traek:  Condition  B 
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Table  18.  Subject’s  Self-Estimation  of  Percent  Time  On-Track:  Condition  C 


Mean 

64 

Std  Dev 

18.257419 

Std  Err  Mean 

3.6514837 

Upper  95%  Mean 

71.536292 

Lower  95%  Mean 

56.463708 

N 

25 

The  information  in  Table  19  shows  the  distribution  of  reporting  on  the  navigation 
evaluator’s  performance  during  Condition  A  and  Condition  B  respectively.  Condition  C 
did  not  use  the  navigation  evaluator.  We  asked  the  subjects  to  rate  the  following  elements 
of  the  navigation  report  on  a  five-point  Likert  scale  where  one  was  “not  at  all  well”  and 
five  was  “very  well”:  timeliness,  clarity,  and  the  content  of  the  information.  The  most 
common  report  for  all  elements  was  a  five.  For  the  case  of  the  study,  this  figure  clearly 
illustrates  the  role  of  navigation  evaluator  was  carried  out  successfully.  Additionally, 
when  asked  if  there  was  anything  the  subjects  would  recommend  adding  to  the  navigation 
report,  several  subjects  provided  the  same  recommendations.  Out  of  the  50  surveys  given 
between  both  conditions;  3  of  50  (6%)  recommended  adding  the  time  until  the  next  turn, 
8  of  50  (16%)  recommended  adding  a  recommended  course  to  regain  track,  and  4  of  50 
(8%)  recommended  reporting  if  the  ship  was  opening  or  closing  onto  track.  The  last 
information  the  opening  or  closing  onto  track  -  represents  the  standard  way  that 
navigation  evaluators  let  the  conning  officer  know  if  they  ship  is  drifting  away  or  drifting 
towards  the  center  of  the  track.  A  breakdown  of  the  response  data  can  be  found  in 
Appendix  L. 
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Navigation  Evaluator  Reports  Evaluation 

Average 

Std.  Dev. 

Mode 

Info  was  Timely:  Condition  A 

4 

1.08 

5 

Info  was  Timely:  Condition  B 

3.92 

0.99 

5 

Info  was  Loud  and  Clear:  Condition  A 

4.52 

0.87 

5 

Info  was  Loud  and  Clear:  Condition  B 

4.44 

0.77 

5 

Necesarry  Info  was  Provided:  Condition  A 

4.36 

0.95 

5 

Necesarry  Info  was  Provided:  Condition  B 

4.16 

0.94 

5 

Table  19.  Evaluations  of  Navigation  Evaluator  Reports 


The  information  in  Table  20  shows  the  results  of  subject  evaluations  of  the  HUD 
components.  We  asked  the  subjects  to  rate  the  following  elements  of  the  HUD  on  a  five- 
point  Likert  scale  where  one  was  “illegible”  and  five  was  “very  legible”:  legibility  of  the 
font,  size  of  the  font,  contrast  of  the  font.  We  asked  the  subjects  to  rate  the  ease  of 
deployment  on  a  five-point  Likert  scale  where  one  “very  difficult”  and  five  was  “very 
easy.”  We  asked  the  subjects  to  rate  the  level  of  distraction  on  a  five-point  Likert  scale 
where  one  “very  distracting”  and  five  was  “not  distracting.”  In  both  conditions  B  and  C, 
the  subjects  rated  all  HUD  elements  above  average.  Additionally,  we  asked  subjects  to 
rate  how  helpful  the  HUD  was  during  the  transit  on  a  five-point  Likert  scale,  where  one 
was  “not  helpful”  and  five  was  “very  helpful.”  Over  90%  of  the  subjects  reported  a  four 
or  five,  indicating  that  they  found  the  HUD  helpful.  A  breakdown  of  the  response  data 
can  be  found  in  Appendix  L. 
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HUD  Evalution 

Average 

Std.  Dev. 

Mode 

Level  of  Distraction:  Condition  B 

4.04 

0.93 

4 

Level  of  Distraction:  Condition  C 

3.96 

1.02 

4,5 

Legibility  of  Font:  Condition  B 

4.32 

0.90 

5 

Legibility  of  Font:  Condition  C 

4.36 

0.81 

5 

Size  of  Font:  Condition  B 

4.44 

0.71 

5 

Size  of  Font:  Condition  C 

4.32 

0.75 

5 

Contrast  of  Font:  Condition  B 

4.40 

0.72 

5 

Contrast  of  Font:  Condition  C 

4.32 

0.75 

5 

Ease  of  HUD  depioyment:  Condition  B 

4.24 

1.12 

5 

Ease  of  HUD  depioyment:  Condition  C 

4.12 

1.12 

5 

Heipfuiness  of  HUD:  Condition  B 

4.72 

0.54 

5 

Heipfuiness  of  HUD:  Condition  C 

4.48 

0.65 

5 

Table  20.  Evaluations  of  the  HUD  Elements 


When  asked  if  the  necessary  information  was  provided  in  the  HUD  to  accomplish 
their  task,  the  subjects  were  presented  a  Likert  scale  where;  one  was  “not  enough 
information,”  three  was  “enough  information,”  and  five  was  “too  much  information.”  The 
surveys  indicated  that  54%  of  the  subjects  reported  having  “enough  information”  (option 
3)  and  that  94%  of  subjects  selected  options  two,  three,  or  four  on  the  above  scale.  This 
indicates  that  the  subjects  were  provided  enough  resources  in  the  HUD  to  accomplish 
their  task. 

The  information  in  Table  21  illustrates  the  breakdown  of  how  subjects  ranked  the 
conditions  and  therefore  the  navigation  information  reporting  methods.  The  question  read 
as  follows  (APPENDIX  E): 

When  you  think  back  about  your  experience,  how  would  you  rank  the 
experimental  setups  used  for  conning  out  of  the  channel?  *Rank  from  one 
to  three.  One  being  the  most  useful  for  staying  on  track,  three  being  the 
least.*  [sessions  to  be  ranked;  Nav  Eval  Only  (A),  Nav  Eval  and  HUD  (B), 
or  HUD  only  (C)] 

Of  the  nine  possible  combinations  of  responses,  only  three  orderings  were 
reported.  Three  subjects  (15%)  ordered  the  setups  as  (1st;  B,  2nd;  A,  3rd;  C),  i.e.,  (1st; 
Nav  Eval  and  HUD,  2nd;  Nav  Eval  Only,  3rd;  HUD  only).  Eight  subjects  (40%)  ordered 
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the  setups  as  (1st:  B,  2nd:  C,  3rd:  A),  i.e.,  (1st:  Nav  Eval  and  HUD,  2nd:  HUD  only,  3rd: 
Nav  Eval  Only).  Nine  of  the  subjects  (45%)  ordered  the  setups  as  (1st:  C,  2nd:  B,  3rd:  A) 
(1st:  HUD  only,  2nd:  Nav  Eval  and  HUD,  3rd:  Nav  Eval  Only).  Of  the  25  subjects,  five 
did  not  report  any  order,  but  only  selected  one  scenario.  Of  those,  two  subjects  (8%) 
selected  only  B,  (Nav  Eval  and  HUD)  and  three  subjects  (12%)  selected  only  C  (HUD 
only). 

Reviewing  these  reports,  and  those  on  the  usefulness  of  the  HUD,  we  find  that  the 
average  reception  of  the  HUD  is  significant  enough  to  be  able  to  make  a  determination  of 
our  secondary  hypothesis.  All  25  officers  in  the  study  rated  one  of  two  conditions  that 
used  HUD  as  their  preferred  choice  they  ranked  condition  B  (Nav  Eval  and  HUD),  or 
condition  C  (HUD)  as  their  1st  choice.  Out  of  20  subjects  who  ranked  all  three 
conditions,  none  of  the  subjects  selected  the  condition  A  (Nav  Eval)  as  their  1st  choice; 
only  3  subjects  selected  it  as  their  2nd  choice;  and  all  other  subjects  selected  it  as  their 
least  preferred  (3rd)  choice.  This  confirms  that  the  subjects  found  HUD  to  be  useful  for 
their  task  performance,  suggesting  their  receptiveness  to  the  idea  of  adding  a  HUD 
concept  to  their  operations.  We  therefore  conclude  that  surface  warfare  officers  will  be 
receptive  toward  the  integration  of  a  similar  system  in  their  daily  operations. 

•  Alternate  Hypothesis:  H2a  The  surface  warfare  officers  will  be  receptive 
to  the  idea  of  integrating  and  using  such  a  system  into  their  daily 
operations 


(C.B.A) 


(CAB)| 


Order 

Count 

Prob 

(B,A,C) 

3 

0.15000 

(C,A,B) 

8 

0.40000 

(C,B,A) 

9 

0.45000  1 

Total 

20  1.00000 

Table  21 .  Subjects’  Preferred  Conditions  (Ordered  Best  to  Worst) 
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The  information  in  Table  22  shows  the  results  of  subjeet  evaluations  of  the 
simulated  environment.  We  asked  the  subjects  to  rate  the  following  elements  on  a  five- 
point  Likert  scale  where  one  was  “inaccurate”  and  five  was  “accurate”:  ship’s  motion 
physics,  visual  representation  of  objects  internal  to  the  ship,  and  visual  representation  of 
objects  external  to  the  ship.  The  most  common  report  for  each  of  these  questions  was  a 
four.  The  average  report  for  ship  motion  physics  was  3.60  (st.  dev.  0.82).  The  average 
report  for  visual  representation  of  objects  internal  to  the  ship  was  3.72  (st.  dev.  0.89).  The 
average  report  for  visual  representation  of  objects  external  to  the  ship  was  3.60  (st.  dev. 
1.08).  Additionally,  we  asked  the  subjects  to  rate  how  well  the  navigation  system  internal 
to  the  ship  allowed  them  to  perform  the  given  task  on  a  five-point  Likert  scale  where  one 
was  “hindered  performance”  and  five  was  “aided  performance.”  The  average  report  for 
this  question  was  3.64  (st.  dev.  1.08)  with  the  mode  being  three.  A  breakdown  of  the 
response  data  can  be  found  in  Appendix  L. 


System  Evalution 

Average 

Std.  Dev. 

Mode 

Ship  Motion  Physics 

3.6 

0.82 

4 

Visuai  Representation  of  objects  inside  the  ship 

3.72 

0.89 

4 

Visuai  Representation  of  objects  outside  the  ship 

3.6 

1.08 

4 

How  weii  did  the  simuiated  movement  (within 
the  ship's  bridge)  ailow  you  to  perform  you  task? 

3.64 

0.99 

3 

Table  22.  Post-Experiment  Evaluation  of  the  VE 


(2)  SSQs 

The  information  in  Table  23  presents  the  results  of  data  reported  on  subjects’ 
symptoms  related  to  SSQs.  There  were  100  reports  in  all,  a  total  of  four  SSQs  per  subject: 
one  SSQ  after  the  training  session  and  then  one  SSQ  after  each  of  the  three  ship-driving 
sessions.  Eor  the  most  part,  the  subjects  reported  having  no  symptoms  of  any  kind. 
Additionally,  not  a  single  subject  reported  having  severe  symptoms  of  any  kind.  The 
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most  common  symptoms  were  reports  of  slight:  eye  strain  (31%),  fatigue  (23%), 
difficulty  focusing  (21%),  and  blurred  vision  (21%).  Moderate  symptoms  were  reported 
21  times  with  diffieulty  focusing  (6%),  eye  strain  (5%),  and  dizziness  with  eyes  closed 
(4%)  being  the  most  reported.  Of  the  baseline  SSQs,  11  of  25  subjects  (41%)  reported 
having  some  symptoms  and  of  those  there  were  only  slight  symptoms.  The  most  common 
baseline  reports  were  of  slight:  fatigue  (28%),  eye  strain  (20%),  general  discomfort 
(12%),  and  headache  (12%).  Other  than  what  has  been  reported  above,  there  were  not 
signihcant  differences  from  the  baseline  reports. 
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None 

Slight 

Moderate 

Severe 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

General  discomfort 

83 

83.0% 

16 

16.0% 

1 

1.0% 

0 

0.0% 

Fatigue 

75 

75.0% 

23 

23.0% 

2 

2.0% 

0 

0.0% 

Headache 

95 

95.0% 

5 

5.0% 

0 

0.0% 

0 

0.0% 

Eye  strain 

64 

64.0% 

31 

31.0% 

5 

5.0% 

0 

0.0% 

Difficulty  focusing 

73 

73.0% 

21 

21.0% 

6 

6.0% 

0 

0.0% 

Salivation  increasing 

99 

99.0% 

1 

1.0% 

0 

0.0% 

0 

0.0% 

Sweating 

88 

88.0% 

12 

12.0% 

0 

0.0% 

0 

0.0% 

Nausea 

92 

92.0% 

8 

8.0% 

0 

0.0% 

0 

0.0% 

Difficulty  concentrating 

92 

92.0% 

7 

7.0% 

1 

1.0% 

0 

0.0% 

"Fullness  of  the  head" 

91 

91.0% 

7 

7.0% 

2 

2.0% 

0 

0.0% 

Blurred  vision 

78 

78.0% 

21 

21.0% 

1 

1.0% 

0 

0.0% 

Dizziness  with  eyes  open 

89 

89.0% 

11 

11.0% 

0 

0.0% 

0 

0.0% 

Dizziness  with  eyes  closed 

92 

92.0% 

4 

4.0% 

4 

4.0% 

0 

0.0% 

Vertigo 

96 

96.0% 

4 

4.0% 

0 

0.0% 

0 

0.0% 

Stomach  awareness 

93 

93.0% 

7 

7.0% 

0 

0.0% 

0 

0.0% 

Burping 

97 

97.0% 

3 

3.0% 

0 

0.0% 

0 

0.0% 

Table  23.  SSQ  Results  from  Study 
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3. 


Post-Session  Interviews 


We  conducted  an  interview  with  each  of  the  25  subjects  after  the  completion  of 
their  third  session.  While  we  made  every  attempt  to  ask  the  same  type  of  questions  to 
each  subject,  in  order  to  allow  the  subject  to  freely  express  their  personal  opinions  of  the 
experience,  each  interview  had  a  segment  that  was  unique  to  that  subject.  Given  this  fact, 
we  also  collected  opinions  on  issues  that  subjects  volunteered  to  provide  themselves  (not 
all  subjects  commented  all  issues,  and  so  a  standard  statistical  analysis  is  not  applicable 
to  this  data  set).  The  following  text  provides  a  breakdown  of  the  more  frequent  responses 
received. 

One  of  the  most  commonly  shared  opinions  was  that  the  shipboard  experience 
was  fairly  realistic.  Ten  subjects  cited  that  either  the  ships  physics  system  or  the  overall 
simulator  was  good.  The  most  common  complaint  against  the  system,  shared  by  seven 
subjects,  was  that  the  ship  handled  too  responsively.  This  meant  that  the  subjects  felt  the 
ship  turned  far  quicker  (sharper)  than  expected.  Two  of  the  subjects  reported  the  system 
to  be  competitive  or  “on-par”  with  the  COVE  simulator.  Another  topic  that  saw 
frequency  in  the  responses  was  the  area  of  moving  around  the  bridge.  Whereas  four 
subjects  reported  that  the  four  predetermined  positions  offered  enough  freedom  of 
mobility,  another  four  subjects  would  have  liked  to  have  had  a  centerline  position 
available.  Two  subjects  felt  the  movement  from  one  position  on  the  ship  to  another 
position  was  too  slow. 

When  asked  about  the  HUD,  the  most  common  opinion  disclosed  (five  subjects) 
was  that  the  subjects  were  able  to  clearly  see  “through”  the  augmented  layer  and  still  take 
in  the  visual  information  outside  (3D  environment  representing  the  ship,  water  and 
terrain).  Three  subjects  reported  that  the  text  font  in  augmented  layer  (HUD)  was  too  big. 
This  was  an  early  realization  of  the  design  team;  however,  the  size  of  the  text  was 
dictated  by  the  low  resolution  of  the  HMD.  Five  subjects  found  the  HUD  to  be  distracting 
but  two  of  those  subjects  claimed  that  it  was  only  initially  so. 

Due  to  the  networked  nature  of  the  simulation  design  there  were,  at  times,  slight 
differences  in  what  the  conning  officer  might  have  seen  in  the  HUD  and  what  the 
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navigation  evaluator  was  reporting  on  aurally.  Five  of  the  subjects  reported  noticing  the 
difference  in  what  was  being  reported  aurally  and  what  they  saw  visually.  Three  subjects 
said  they  would  have  placed  more  trust  in  the  HUD  whilst  two  subjects  claimed  they 
would  have  put  their  trust  in  what  they  would  hear  from  the  navigation  evaluator. 

The  most  commonly  answered  question  was  what  a  user’s  preferred  condition 
was.  Eight  of  the  subjects  reported  that  they  preferred  the  condition  with  only  the  HUD. 
12  of  the  subjects  reported  that  they  preferred  the  condition  with  the  navigation  evaluator 
and  the  HUD.  Four  subjects  reported  being  “comfortable”  with  the  redundancy  of  having 
the  navigation  evaluator  reports.  Four  subjects  also  reported  feeling  hesitant  or  bad  about 
cutting  off  the  navigation  evaluator  while  he  was  making  reports.  The  overall  reporting  in 
the  interviews  did  not  match  the  final  questionnaire  data  when  it  came  to  preference  of 
condition.  A  reason  for  this  may  be  that,  although  subjects  felt  they  performed  better  with 
only  the  HUD,  they  may  simply  have  preferred  having  the  navigation  evaluator  as  well. 

4,  Behavioral 

We  monitored  all  75  sessions  for  physical  behavioral  cues  -  we  made  video 
recording  of  all  sessions  (the  camera  view  captured  subjects  as  they  were  standing  and 
moving  in  front  of  the  workstation).  While  a  few  subjects  displayed  slight  disorientation 
when  first  donning  the  headset,  no  subject  exhibited  the  signs  of  mild  or  severe  dizziness, 
fell,  or  injured  themselves  during  the  study.  All  cues  of  physical  behaviors  witnessed 
during  the  sessions  were  on  par  for  what  we  would  expect  a  real-world  conning  officer  to 
display.  There  were  no  significant  behavioral  data  to  report  on. 

E.  SUMMARY 

This  chapter  discussed  the  user  study  and  accompanying  results.  The  data 
included  information  about  subjects’  demographics,  performance  data,  self-reported  data 
(surveys  and  interviews),  and  behavioral  data  (video  recordings).  The  most  important 
findings  of  the  study  were  that  subjects  performed  better  when  using  the  HUD. 
Additionally,  the  issues  related  to  cognitive  tunneling  and  cybersickness  were  minimal. 


no 


VI.  CONCLUSION  AND  RECOMMENDATION 


This  chapter  highlights  the  main  eontributions  of  the  study  and  details  future  work 
that  we  see  as  viable  within  the  domain.  Whereas  the  study  was  initially  proposed  as  a 
means  of  testing  an  operational  eoneept  onboard  U.S.  Navy  ships,  the  end  result  has 
potential  eontributions  to  the  fields  of  military  researeh,  VR,  and  AR.  The  ehapter  lays 
out  guidanee  for  the  eontinuation  of  the  work  presented  in  this  paper  as  well  as 
reeommended  implementations  of  AR  onboard  Navy  ships. 

A,  MAIN  CONTRIBUTIONS 

The  work  of  this  study  eontributes  to  the  several  domains.  Firstly,  in  the  domain 
of  naval  researeh,  we  have  provided  genuine  data  that  illustrates  the  utility  of  integrating 
AR  into  the  everyday  operations  of  U.S.  Naval  personnel.  Chapter  V  elearly  lays  out  the 
numerieal  data  that  proves  the  statistieal  signifieanee  of  the  results  gathered  by  the  study. 
The  analysis  eoneludes  that  the  use  of  the  HUD  made  a  positive  differenee  in  the 
subjeets’  performanees  (our  first  hypothesis).  Additionally,  through  the  subjeetive  results, 
we  witnessed  a  very  high  aeeeptanee  rate  of  the  eoneept  (our  seeond  hypothesis).  This 
proof  of  eoneept  for  the  use  of  an  AR  eoneept  on  the  bridge  of  a  ship  applies  to  not  only 
the  navigation  related  displays  but  also  real-world  taetieal  displays.  In  essenee,  we  have 
paved  the  way  for  further  researeh  that  may  equip  the  sailor  of  the  future  with  AR  heads 
up  displays  for  numerous  everyday  operations. 

In  the  general  domain  of  military  researeh,  the  work  that  went  into  designing  and 
building  the  study  is  highly  modular  and  ean  easily  be  duplicated  in  order  to  test  AR 
hardware  integration  into  a  variety  of  operational  settings.  While  displaying  CNI  was  the 
primary  foeus  of  this  study,  the  display  of  taetieal  information  to  Marines  who  aet  in  an 
operational  environment,  or  the  display  of  improvised  explosive  deviees  to  the  bomb 
teehnieians  are  only  slight  modifieations  to  the  methodology.  This  study  proves  that,  for 
minimal  resourees,  these  operational  eoneepts  ean  be  tested  in  a  safe  and  repeatable 
environment  with  negligible  risk  to  the  users. 


Ill 


In  the  domain  of  VR,  this  study  represents  one  of  the  first  researeh-based  uses  of 
an  entirely  new  generation  of  inexpensive  VR  headsets  (the  Oeulus  Rift  used  in  this  study 
is  one  example  of  this  line  of  produets).  What  has  started  to  beeome  a  phenomenon,  the 
produetion  of  consumer  based  VR  hardware  is  taking  off  In  only  the  past  year,  Oeulus, 
Samsung,  Sony,  Zeiss  Optics  and  HTC  partnered  with  Valve/Steam  have  all  created 
prototype  VR  hardware  that  they  plan  to  bring  to  market  soon.  This  work  contributes  to 
the  field  by  taking  great  strains  to  precisely  detail  the  processes  needed  to  create  a  VE 
that  is  implementable  on  the  new  generation  of  hardware.  With  the  entry  price  for  VR 
becoming  so  low,  many  research  projects  outside  the  domain  of  computer  science  can 
now  be  accomplished  in  the  safety  of  a  VE.  This  work  will  contribute  to  that  cause  by 
being  a  great  starting  resource  for  those  looking  to  build  VR  based  studies. 

This  study  has  contributed  to  the  domain  of  AR  by  not  only  finding  a  novel  use 
for  the  technology  in  the  military  domain  but  also  by  modeling  a  method  of  testing  AR 
concepts  in  a  full  VR  simulation.  As  detailed  in  Chapter  II,  AR  is  not  a  new  technology 
in  the  military  domain;  however,  the  NAVHUD  system  is  a  unique  implementation  of 
AR  concept  in  the  domain  of  shipboard  navigation.  The  greater  contribution  to  the  AR 
domain  is  that  we  took  an  AR  concept  and  rigorously  tested  it  in  a  VR  simulation.  The 
reason  this  is  a  better  route  is  that  VR  is  faster  and  cheaper  to  program  for;  especially  if 
control  of  the  external  environment  is  a  critical  factor.  In  this  case,  where  the  requirement 
would  be  to  physically  outfit  a  naval  ship  with  the  system  (and  to  repeatedly  navigate  a 
predefined  course)  to  test  its  usefulness,  we  can  clearly  demonstrate  the  immediate 
benefits  of  testing  this  AR  system  in  VR. 

B,  FUTURE  WORK 

As  this  study  was  designed  to  test  the  operational  usefulness  of  a  HUD  for 
conning  officers,  the  most  obvious  future  work  would  be  a  full-scale  implementation  of 
the  system  with  real  AR  hardware  integrated  into  the  ship’s  current  electronic  navigation 
system.  Of  all  the  positive  feedback  that  the  concepts  incorporated  in  this  study  have 
garnered,  the  most  common  recommendation  given  has  been  that  this  not  be  limited  to 
navigation  operations.  Instead,  many  have  recommended  a  large  scale  AR  application 
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that  integrates  voyage  management,  radar  systems,  and  weapon  systems.  The  holistie 
adoption  of  AR  onboard  ships  would  benefit  all  departments  from  engineering  to 
information  systems.  From  shipboard  firefighting  to  real-time  missile  traeking,  there  is 
room  for  AR  in  all  naval  domains. 

On  the  smaller  seale,  the  next  praetieal  step  from  this  work  would  be  to  take  the 
U.S.  Navy’s  highest  fidelity  ship  driving  simulator,  COVE,  and  test  out  an  AR 
applieation  similar  to  NAVHUD  on  it.  Given  the  eombination  of  the  additional  sensor 
data  that  is  built  into  COVE,  the  validation  of  its  physios  system,  and  the  hundreds  of 
surface  warfare  officers  who  use  the  system  every  year;  a  large  scale  study  (with  COVE 
as  the  base  platform)  would  provide  indisputable  data  set  on  whether  or  not  an  AR 
system  for  conning  officers  is  a  viable  option.  If  that  proves  successful,  the  next  step 
would  be  going  to  a  ship  with  an  AR  headset  and  really  testing  it  in  an  operational 
environment. 

C.  SUMMARY 

With  25  highly  specialized  subjects,  nearly  20  hours  of  ship  driving  performance 
data,  and  hundreds  of  subjective  data  points;  this  study  has  been  a  great  success.  The 
results  provided  in  Chapter  V  firmly  suggest  that  a  lightweight  glasses-type  AR  overlay 
system  for  a  conning  officer  will  increase  his  ability  to  maintain  a  close  proximity  to  a 
preplanned  track.  This  chapter  has  discussed  the  many  contributions  that  this  work  will 
provide  to  the  domains  of  AR,  VR,  naval  research,  and  military  research  in  general. 
Einally,  this  chapter  lays  forth  recommended  continuations  of  this  research,  as  well  as 
similar  veins  of  work  that  can  be  accomplished  in  the  shipboard  AR  domain. 
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APPENDIX  A.  RECRUITMENT  POSTER 


Volunteer  Conning  Officers  needed  for  research  study. 
Help  us  investigate  a  brand  new  way  to  drive  ships 


Your  task  will  be  very  simple.  You  will: 

(1)  Ccxm  a  ship  in  a  virtual  channel  using  the  new  Oculus  Rift™ 

(2)  Be  asked  to  reflect  chi  your  experience. 


WHERE:  SWOS.  Building 
and  room  TBD 
HOW  LONG:  1  x  90  min 
or  2  X  45  min 

WHEN:  December 
through  the  5“  - 
contact  us  to  schedule 
time  (evening  sessions 
for  students!) 

WHO:  Persons  with 
experience  conning  a 
US  naval  vessel. 


Please  contact  LT  Brendan  Geoghegan,  USN  bjgeoQhe@nDs.edU|,  757*597''5131  to  confirm  your  participation  -  give  us  several  options  for  date(s)  &  time{s)  when 
you  will  be  available.  For  any  amplifying  information  on  the  study  please  contact  the  Principal  Investigator,  Dr.  Amela  Sadagic,  (831)  656-3819, 
asadagic@nps.  edu. 


This  study  is  a  part  of  research  being  conducted  by  the  Naval  Postgraduate  Sdiool,  Monterey  CaltfbnHa.  Instbitiona)  review  board  (IRB)  dtair's  contaa  information:  Dr.  Larry  Shattuck,  S314S6-2473,  lashattijOnps.edu 
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APPENDIX  B.  PRE-EXPERIMENT  SURVEY 


Page  1  of  4  » 


Pre-Experiment  Survey  Questions: 

Personal  Background: 

Q:  What  is  your  age?  _ 

Q  What  is  your  sex?  _ 

Professional  Background: 

Q:  Current  Designation?  _ 

Q:  Current  rank?  _ 

Q:  How  many  years  do  you  have  on  active  duty  in  the  USN? _ 

Q:  How  many  years  do  you  have  you  served  as  an  111X  or  116X{SWO)? 

Q:  How  long  has  it  been  since  the  last  time  you  stood  the  watch  of  conning 
officer  (years  /  months)? 


Q:  How  long  has  it  been  since  the  last  time  you  stood  the  watch  of  conning 
officer  during  a  restricted  water  transit  (years  /  months)?  Ex.  Sea  and 
Anchor. 


Q:  How  many  times  have  you  have  conned  or  served  as  OOD 
entering/leaving  port.  (Estimate). 


Q:  Have  you  ever  been  qualified  as  a  Navigator  or  Navigation  Evaluator? 
□Yes  GNo 

If 'Yes' how  long  ago  was  it?  _ (years) 
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Q:  Have  you  ever  served  on  a  fully  ECDIS-N  certified  ship  (lAW  NAVDORM)? 
□Yes  GNo 

If 'Yes' how  long  ago  was  it?  _ (years) 

Q:  How  would  you  rate  your  current  conning  skill  level? 

Bottom  6%  50-36%  75-50%  95-75%  Top  5% 

□  □ODD 

Environment  Questions; 

1 )  What  types  of  video  games  do  you  play?  What  device  do  you  use  to  play  the 
games,  and  how  often  do  you  play  them? 

Do  you  play  games  at  all? 


O  NO -go  to  question  #2 
O  YES  -  answerthe  following  questions: 


Type  of  Game 

Devices 

How  often? 

(check  all  that 
apply) 

(Check  all  that  apply) 

(Check  one  and  then  enter  your  usage  hours.) 

First  Person 

1  play  them  on  (check^that  apply): 

Hours  per  day 

Hours  per  week 

Hours  per  month 

Rarely 

Shooter  O 

Computer  0 

Smartphone  O 

Tablet,  Ipad  O 

GameConsole  0 

O 

O 

0 

O 

Other  Cellphone  O 

E-Reader  O 

Enter  #  of 

Enter  #  of 

Enter  #  of 

hours; _ 

hours; _ 

hours: _ 

1  play  them  on(check  all  that  apply): 

Hours  per  day 

Hours  per  week 

Hours  per  month 

Rarely 

Computer  O 

Smartphone  O 

Tablet.  Ipad  O 

GameConsole  O 

0 

O 

O 

O 

Multiplayer 

Other  Cellphone  O 

E-Reader  O 

Enter  #  of 

Enter  #  of 

Enter  #  of 

games 

hours: _ 

hours: _ 

hours: _ 

1  playthemon(check  all  that  apply): 

Hours  per  day 

Hours  per  week 

Hours  per  month 

Rarely 

Adventure.  0 

Computer  O 

Smartphone  O 

Fantasy, 

Tablet.  Ipad  O 

GameConsole  O 

O 

O 

O 

O 

Role  Playing 

Other  Cellphone  O 

E-Reader  O 

Enter  #  of 

Enter  #  of 

Enter  #  of 

hours; 

hours; 

hours; 
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1  play  them  on(check  ail  that  apply): 

Hours  per  day 

Hours  per  week 

Hours  per  month 

Rarely 

Other 

O 

Computer  O 

Tablet.  Ipad  O 

Smartphone  O 

Game  Console  O 

0 

O 

O 

O 

games 

Enter  thegenre: 

Other  Cellphone  O 

E-Reader  O 

Enter  #  of 
hours; _ 

Enter  #  of 
hours; _ 

Enter  #  of 
hours; _ 

2)  Were  you  required  to  use  training  simuiations  or  simuiators  at  any  point  in 
your  career?  (exampies:  COVE  or  Fiight  Sims.) 

□Yes  GNo 

O  NO- go  to  question  #3 
O  YES  -  answerthe  following  questions: 

a.  Enter  the  names  of  those  simulations,  what  skills  were  they  used  to  train, 
how  many  hours  of  training  in  total,  and  the  date  of  last  usage?  Note***  If 
you  do  not  remember  the  name  of  the  simulation,  then  please  enter  its 
ciosest  description  instead. 

1 .  Simuiation  #1 : _ 

Skiiis: _ 

Total  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 

2.  Simuiation  #2: _ 

Skiiis: _ 

T otal  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 

3.  Simuiation  #3: _ 

Skiiis: _ 

Total  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 
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4.  Simulation  #4: _ 

Skills: _ 

T otal  number  of  hours  (approximate): _ 

Date  of  last  use  (approximate):  _ 

3)  Have  you  ever  used  a  Head  Mounted  Display  with  motion  tracking? 

□Yes  GNo 

4)  Have  you  ever  used  an  augmented  reality  simulation?  (ex.  Google  Glass, 
Aviation  Helmet  HUD) 

□Yes  GNo 
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APPENDIX  C.  CONSENT  FORMS 


A,  IRB  CONSENT 


Page  1  of  2 


Naval  Postgraduate  School 
Consent  to  Participate  in  Research 

Introduction.  You  are  imited  to  participate  in  a  research  stud>'  conducted  by  the  Na\-al  Postgraduate 
School  entitled  Na\igational  Heads  up  Disfday  (NAXHUD):  will  a  shipboard  augmented  electrcoic 
na\igation  system  sink  or  swim.  The  purpose  of  the  research  is  to  determine  the  viability  of  an  operational 
setup  of  the  bridge,  onboard  US  Na\'y  w'arships,  in  which  a  conning  officer’s  visual  fieldis  augmented  with 
an  overlay  of  critical  naMgational  infcimaticn  (CNI)diat  is  typically  relayed  in  aural  form  frcm  the 
navigation  officer. 

Procedures. 

You  will  be  asked  to  run  through  a  short  virtual  reality  tutorial.  Youwill  then  be  briefed  on  a 
transit  plan  out  of a  charnel.  You  will  then  com  a  ship  out  of a  virtual  charnel  using  standard 
commands.  Finally  you  will  be  asked  to  evaluate  the  heads  up  di^lcty  system. 

This  study  will  take  cpproximately  one  hour  and  thirty  minutes. 

You  will  perform  the  task  alone.  There  will  be  a  toted  of  approximately  40  participants. 

You  will  take  part  in  three  experimental  sessions,  om  with  a  navigation  evaluator,  onewith  a 
navigation  evaluator  and  a  head’s  updisplay,  andonewith  Just  the  head's  up  display. 

Your  participation  is  helping  acquire  neyv  understandings  on  use  of  heads  updisple^-  technology  in 
shipboardnavigation. 

Audio  andvideo  of  your  experience  aswellas  a  recording  of  your  virtual  experience  will  be  taken  for 
post  experiment  analysis. 

Location.  The  inteniew'  surv  ey  experiment  will  take  place  in  an  office  space  in  the  International  studies 
u  at  the  Surface  Warfare  Officer  School,  Newport  R1  or  in  an  office  space  in  Watkins  Hall  at  Naval 

Postgraduate  School,  Monterey  CA. 

Cost.  There  is  no  cost  to  participate  in  this  research  study. 

Vohintarj' Nature  of  the  Studj-.  Yourpartidpationinthis  study  is  strictly  voluntary.  If  you  choose  to 
participate  you  can  change  your  mind  at  any  time  and  withdraw'  from  the  study.  Y  ou  will  not  be  penalized 
in  any  w'ay  or  lose  any  benefits  to  w'hidi  you  w'ould  otiierwise  be  entided  if  you  choose  not  to  parti  dpate  in 
this  study  or  to  withdraw'.  The  alternative  to  participating  in  theresearch  is  to  notpartidpate  in  thereseardL 

Potential  Risks  and  Discomforts.  The  potential  risks  ofparticipaiing  in  this  study  are:  ^’hile  every  effort 
in  the  design  of  the  virtual  environment  testi?ig  platform  has  been  made  to  mitigate  cyber  sickness,  there 
is  a  possibility  the  subject  mt^'  have  synptoms  present  during  the  study.  Symptoms  include  visual 
sy  mptoms  ^eyestrains,  blurredvision,  headaches),  disorientation  (vertigo,  imbalance)  and  nausea 
(vomiting,  dizziness). 

Anticipated  Benefits.  Antidpated  benefits  fi^omthis  study*  are  experiment  will  benefitthe  USNavy  in 
terms  of  savings  through  reduction  of  errors,  increased  precision  in  execution  of  ship  navigation,  and  a 
potential  savings  through  reduced  mcoming.  You  will  not  directiy  benefit  from  your  parti  dpation  in  this 
research. 

Vision  w 
Date: 


Jj 
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CompensatioD  for  Paitidpation.  No  tangible  compensaticnwill  be  given. 

Confidentialit}'  &  Prhao'  Act.  Any  in  formation  that  is  obtained  during  this  studv'  willbekept 
confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within  reason,  will  be  made  to  keep  your 
personal  infcmiation  in  your  researdi  recwd  confidaitial  but  total  confidentiality  cannot  be  guaranteed. 

No  information  will  be  pubUdy  accessible  which  could  identify  me  as  a  participant.  You  will  be 
identified  only  as  a  code  number  on  all  researdi  fcxms  databases;  yournameon  any  signed  document 
will  not  be  paired  withmy  code  number  in  order  to  protect  your  identity.  Youxmderstand  that  records  of 
your  partidpatioa  will  be  maintained  by  NPS  forten  years.  How’ever,  it  is  possiblediat  the  researcher 
maybe  required  to  dhnlge  information  obtained  in  the  course  of  diis  research  to  file  subject’s  chain  of 
command  or  other  legal  body. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  researdi,  or  you  experience  an  injury 
or  have  questions  about  any  discomfOTts  diat  you  experience  w'hile  taking  part  in  this  smcfyplease  contact 
the  Principal  Investigator,  Dr  Ame/aSadagic,  (851)  656-5819,  asadQgic@ips.edu,  Questions  about  your 
rights  as  a  research  subject  or  any  other  concerns  may  be  addressed  to  the  Navy  Postgraduate  School  IRB 
Chair,  Dr.  Lany-  Shattuck,  83 1-656-2473,  kshattufSnps.edu. 

Statement  of  Consent.  IhavereadtiieinformaticHipro\ided  abo\  e.  I  have  been  given  the  oppOTtunityto 
ask  questions  and  all  the  questions  have  been  answ'eredto  my  satisfaction.  Ihavebeenpro\ideda  copyof 
this  form  formy  records  and  I  agree  to  participate  in  this  study.  I  undentand  diat  by  agreeing  to 
participate  in  this  research  and  signing  this  form,  I  do  not  w'aive  any  of  my  legal  rights. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 


B 


AUDIO  AND  VISUAL  CONSENT 
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Piivacy  act  Statement  and  Consent  Agreement 
foi  Audio  or  Mdeo  Recoiding 


I  have  received  a  thorough  description  of  the  purpose  and  procedures  for  video  recording 
during  the  course  of  the  proposed  research  study.  I  give  my  consent  to  allow  recording 
during  participation  in  this  study,  and  for  those  records  to  be  reviewed  by  persons 
involved  in  the  study,  as  well  as  used  to  produce  a  material  (images  or  videos)  that  may 
be  used  to  present  the  results  of  this  study  at  conferences,  in  research  papers  and 
professional  research  meetings.  I  understand  that  all  information  will  be  k^t  confidential 
and  will  be  reported  in  an  anonvmous  fashion,  and  that  the  recordings  will  be  erased  five 
(5)  years  after  the  study  has  been  completed.  I  further  understand  that  I  may  withdraw' 
this  consent  at  any  time  without  penalty. 


Video  recordings  will  be  used  to  review  physical  cues  of  the  subject  w'hile  conning. 
These  may  include  number  of  times  the  HUD  is  deployed,  where  the  subject  is 
preeminently  looking,  posture  indications,  simulator  sickness  svmptoms,  etc. 

Audio  recordings  will  be  used  to  review  aural  cues  such  as  conning  officer  navigator 
interaction,  misheard  reports  to  the  helm,  number  of  commands  given,  etc. 


Participant's  Signature 


Date 


Researcher's  Signature 


Date 
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APPENDIX  D.  SIMULATOR  SICKNESS  QUESTIONNAIRE 


No 


Date 


SIMULATOR  SICKNESS  QUESTIONNAIRE 

Kennedy,  Lane,  Berbaum.  &  Lilienthal  (1993)*** 


Instructions  :  Circle  how  much  each  svmotom  below  is  affectina  vou  right  now. 

1 .  General  discomfort 

None 

Slight 

Moderate 

Severe 

2.  Fatigue 

None 

Slight 

Moderate 

Severe 

3.  Headache 

None 

Slight 

Moderate 

Severe 

4.  Eye  strain 

None 

Slight 

Moderate 

Severe 

5.  Difficulty  focusing 

None 

Slight 

Moderate 

Severe 

6.  Salivation  increasing 

None 

Slight 

Moderate 

Severe 

7.  Sweating 

None 

Slight 

Moderate 

Severe 

8.  Nausea 

None 

Slight 

Moderate 

Severe 

9.  Difficulty  concentrating 

None 

Slight 

Moderate 

Severe 

10.  «  Fullness  of  the  Head  » 

None 

Slight 

Moderate 

Severe 

1 1 .  Blurred  vision 

None 

Slight 

Moderate 

Severe 

12.  Dizziness  with  eyes  open 

None 

Slight 

Moderate 

Severe 

13.  Dizziness  with  eyes  closed 

None 

Slight 

Moderate 

Severe 

14.  *  Vertigo 

None 

Slight 

Moderate 

Severe 

15.  **Stomach  awareness 

None 

Slight 

Moderate 

Severe 

16.  Burping 

None 

Slight 

Moderate 

Severe 

*  Vertigo  is  experienced  as  loss  of  orientation  with  respect  to  vertical  upright. 

**  Stomach  awareness  is  usually  used  to  indicate  a  feeling  of  discomfort  which  is  just  short  of 
nausea. 


Last  version  :  March  2013 

•••Original  version  :  Kennedy.  R-S.,  Lane,  N.E.,  Berbaum,  K.S.,  &  Liiienlha!.  M.G.  ( 1993).  Simulator  Sickness 
Questionnaire:  An  enhanced  method  for  quantifying  simulator  sickness.  Intemalionai  Journal  of'Aviaiion  Psychology'. 
i(3),  203-220. 
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APPENDIX  E.  EXPERIMENT  SURVEYS 


A,  NAVIGATION  EVALUATOR  ONLY 


Page  I  of  1 


Performance  Questions  (Only  the  auditory  Nav  Eval  inputs); 


Q:  How  much  of  the  time  do  you  feel  you  were  on  track?  (<10  yds  offtrack) 

20%  of  time  40%  60%  80%  1 00%  of  time 

□  □  □  □  □ 


Q:  Looking  back  please  evaluate  the  reports  given  by  the  Navigation  Evaluator. 


Information  was  timely: 

1  (not  at  all  well)  2  3  (average)  4 

□  □  □  □ 

Information  was  loud  and  clear: 

1  (not  at  all  well)  2  3  (average)  4 

□  □  □  □ 

Necessary  information  was  provided  for  the  task  given: 
1  (not  at  all  well)  2  3  (average)  4 

□  □  □  □ 


5  (very  well) 

□ 


5  (very  well) 

□ 


5  (very  well) 

□ 


Was  there  any  additional  reported  information  you  would  have  liked  to  receive 
from  the  Navigation  Evaluator? 
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B.  NAVIGATION  EVALUATOR  AND  HUD 


Page  1  of  2  » 


Performance  Questions  (HUD  and  Nav  Eval): 

Q:  How  much  of  the  time  do  you  feel  you  were  on  track?  (<10  yds  offtrack) 

20%  of  time  40%  60%  80%  100%  of  time 

□  □  □  □  □ 

Q:  Looking  back  please  evaluate  the  reports  given  by  the  Navigation  Evaluator. 
Information  was  timely: 

3  (average) 


1  (not  at  all  well) 

□ 


2 

□ 


□ 


4 

□ 


Information  was  loud  and  clear: 

1  (not  at  all  well)  2 

□  □ 

Necessary  information  was  provided  for  the  task  given: 
1  (not  at  all  well)  2  3  (average)  4 


3  (average) 

□ 


4 

□ 


□ 


□ 


□ 


□ 


5  (very  well) 

□ 


5  (very  well) 

□ 


5  (very  well) 

□ 


Was  there  any  additional  reported  information  you  would  have  liked  to  receive 
from  the  Navigation  Evaluator? 


Q:  Thinking  back  to  your  session,  please  evaluate  the  HUD  (visual  overlay 
consisting  of  the  navigational  information). 


Level  of  distraction: 
1  (very  distracting)  2 


□ 


□ 


3 

□ 


Legibility  of  the  textual  reports: 
Font: 


1  (illegible) 

□ 

Size: 
1  (illegible) 
□ 


Contrast: 


1  (illegible) 

□ 


2 

□ 


2 

□ 


2 

□ 


3  (average) 

□ 


3  (average) 

□ 


3  (average) 

□ 


4  5  (not  distracting) 

□  □ 


4  5  (very  legible) 

□  □ 


4 

□ 


4 

□ 


5  (very  legible) 

□ 


5  (very  legible) 

□ 
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Necessary  information  was  provided  for  the  task  given: 


1  (not  enough 

2 

3  (enough 

4 

5  (too  much 

information) 

information) 

information) 

□ 

□ 

□ 

□ 

□ 

Ease  of  depioymentof  the  HUD  (Tapping  the  Hud): 

1  (very  difficuit) 

2 

3  (average) 

4 

S  (very  easy) 

□ 

□ 

□ 

□ 

□ 

How  heipfui  was  the  HUD: 

1  (not  heipfui) 

2 

3  (average) 

4 

5  (very  helpful) 

□ 

□ 

□ 

□ 

□ 
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C.  HUD  ONLY 


Page  1  of  I  ♦ 


Performance  Questions  (HUD  only): 

Q:  How  much  of  the  time  do  you  feel  you  were  on  track?  (<10  yds  off  track) 


20%  of  time 

40% 

60% 

80% 

1 00%  of  time 

□ 

□ 

□ 

□ 

□ 

Thinking  back  to  yoursession,  please  evaluate  the  HUD  (visual  overlay 

consisting  of  the  navigational  information). 

Level  of  distraction: 

1  (very  distracting) 

2 

3 

4 

5  (not  distracting) 

□ 

□ 

□ 

□ 

□ 

Legibility  of  the  textual  reports: 

Font: 

1  (illegible) 

2 

3  (average) 

4 

5  (very  legible) 

□ 

□ 

□ 

□ 

□ 

Size: 

1  (illegible) 

2 

3  (average) 

4 

5  (very  legible) 

□ 

□ 

□ 

□ 

□ 

Contrast: 

1  (illegible) 

2 

3  (average) 

4 

5  (very  legible) 

□ 

□ 

□ 

□ 

□ 

Necessary  information  was  provided  for  the  task  given: 

1  (not  enough 

2 

3  (enough 

4 

5  (too  much 

information) 

information) 

information) 

□ 

□ 

□ 

□ 

□ 

Ease  of  deployment  of  the  HUD  (Tapping  the  Hud): 

1  (very  difficult) 

2 

3  (average) 

4 

5  (very  easy) 

□ 

□ 

□ 

□ 

□ 

How  helpful  was  the  HUD: 

1  (not  helpful) 

2 

3  (average) 

4 

5  (very  helpful) 

□ 

□ 

□ 

□ 

□ 
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D.  POST  EXPERIMENT 


Pagel  of  1  ” 


Post  Experiment  Survey: 


Q;  When  you  think  back  about  your  experience,  how  would  you  rank  the 
experimental  setups  used  for  conning  out  of  the  channel?  'Rank  from  one  to 
three.  One  being  the  most  useful  for  staying  on  track,  three  being  the  least' 

NavEvaiOnly  NavEval  and  HUD  HUD  only 

□  □  □ 


Q:  How  well  did  you  feel  the  virtual  environment  accurately  portrayed  the 
experience  of  conning  out  of  a  harbor? 

Ship  motion  physics  were: 

1  (not  at  all  accurate)  2  3  4  5  (very  accurate) 

□  □  □  □  □ 

Visual  representation  of  expected  objects  within  the  ship: 

1  (not  at  all  accurate)  2  3  4  5  (very  accurate) 

□  □  □  □  □ 

Visual  representation  of  expected  objects  outside  the  ship: 

1  (not  at  all  accurate)  2  3  4  5  (very  accurate) 

□  □  □  □  □ 


How  well  did  the  simulated  movement  (within  the  ship’s  bridge)  allow  you 
to  perform  you  task? 


1  (it  hindered 
performance) 

□ 


2 

□ 


3  (it  had 
no  effect) 

□ 


4  5  (it  aided 

performance) 

□  □ 
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APPENDIX  F.  EXPERIMENT  CHECKLIST 


Page  1  of  1 


EXPERIMENT  CHECKLIST: 


SUBJECTID 


Welcome  the  subject  and  thank  them  for  participating 

IRB  Informed  Consent 

Audio/Visual  Consent 

Pre-Experiment  Survey 

Baseline  SSQ  (No.l) 

Training  Simulation 

SSQ(No.2) 

Ship  characteristics  sheet  (keep  there) 

1  nstructions  A/B/C 

Simulation  A/B/C 

SSQ{No.3) 

Instructions  A/B/C 

Simulation  A/B/C 

SSQ{No.4) 

Instructions  A/B/C 

Simulation  A/B/C 

SSQ(No.5) 

Post-Experiment  Survey 

Debriefing  Script 

Thank  them  for  volunteering  and  offerthem  a  chance  at  the  other  oculus. 
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APPENDIX  G.  TASK  INFORMATION  SHEET 


Page  1  of  1  • 


Welcome, 

You  will  beaskedto  conn  a  naval  vesseloutofa  channel.  Please  revlewthe  following  ship 
specifications  and  navigation  plan  fordetails  of  the  transit.  Once  the  simulation  begins  you  will  not  have 
immediate  access  to  this  data,  so  please  study  it  carefully.  Feelfreetoaskany  questions. 

Vessel: 

Length:  400  Ft: 

Width:  60ft 
Draft:  20  ft 
Top  Speed:  21  kts 

Engine  Orders: 

(There  will  be  no  engine  orders  given;  the  ship  will  be  set  to  travel  at  an  average  speed  of  15  kts.) 

Rudder  Orders: 

Any  standard  naval  ruddercommands  can  be  used. 

Feelf  ree  to  drive  by  rudder  or  course. 

Bridge  Navigation: 

Just  as  in  yourtraining  tutorial,  in  orderto  move  to  different  positions  on  the  bridge  you  just 
need  to  ask  the  operator/heimsman  to  move  you.  You  can  move  to  left/right  inner  bridge  and  ieft/right 
bridgewings. 

Transit  Plan: 

The  following  is  a  chart  of  the  channel  as  well  as  planned  course. 

All  turns  calculated  for  standard  rudder  at  15  kts  _ 


ISLA  VERDE 

J70* 

IMOyds 

iOOr 

>soo  W,  * 

i  • 

% 

1000  v* 
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APPENDIX  H.  TASK  INSTRUCTIONS 


A,  NAVIGATION  EVALUATOR  ONLY 


Page  I  of  1 


Instructions:  (AUDIO  ONLY) 

We  will  assist  you  in  puttingon  the  head  mounted  display.  When  the  simulation  starts  you  will 
be  placed  at  the  left  center  of  the  bridge.  The  ship  will  be  moving  in  the  water  at  15  knots  heading  in 
the  direction  of  the  first  leg.  As  soon  as  you  are  ready  to  start,  simply  start  giving  commands  to  the 
helmsman  and  try  to  drive  as  close  to  track  as  possible.  One  of  the  performance  metrics  of  this  study  is 
distance  offtrack. 

In  this  scenario  you  will  be  given  the  following  aural  reports  by  the  Nav  Eval: 

Navigation  Reports  will  be  made  coming  out  of  turns  and  every  minute  after.  Est.  3  reports 
per  leg 

•  “I  hold  theshipXXX  yard  (left/right)outoftheturn.  CurrentspeedisXXkts.  Current  course 
overground  is  XXX.  Distance  to  turn  is  XXX  yds.  Next  course  is  XXX.  Atthistime, 
Recommend  (come  (Right/Left)  to  regain  track  /  maintain  course  and  speed).  -  out  of  the  turn 

•  "AttimeXXX,!  hold  the  ship  XXX  yards  (left  of/right  of/on)  track.  CurrentspeedisXXkts. 
CurrentcourseovergroundisXXX.  Distance  to  turn  is  XXX  yds.  Next  course  is  XXX.  Atthis 
time,  Recommend  (come  (Right/Left)  to  regain  track  /  maintain  course  and  speed}."  -Every 
minute  in  the  turn 

•  "200  yards  till  turn"  -at  200  yards  to  turn 

•  "100  yards  till  turn" -at  100  yards  to  turn 

•  "Recommend cometocourse XXX"  -at turn 


Good  Luck! 
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NAVIGATION  EVALUATOR  AND  HUD 


Page  1  of  2  *  < 


Instructions:  (AUDlOand  visual) 

We  will  assist  you  in  putting  on  the  head  mounted  display.  When  the  simulation  starts  you  will 
be  placed  at  the  left  center  of  the  bridge  The  ship  will  be  moving  in  the  water  at  15  knots  heading  in  the 
direction  of  the  first  leg.  As  soon  as  you  are  ready  to  start,  simply  start  giving  commands  to  the 
helmsman  and  try  to  drive  as  close  to  track  as  possible.  One  of  the  performance  metrics  of  this  study  is 
distance  offtrack. 

In  this  scenario  you  will  be  given  the  following  aural  reports  by  the  Nav  Eval  as  well  as  be  able  to  use  a 
Heads  Up  Display: 

Reports  will  be  madecomingout  of  turns  and  every  minute  after.  Est.  3  reports  per  leg 

•  "lholdtheshipXXXyard(left/right)outoftheturn.  CurrentspeedisXXkts.  Current  course 
overground  is  XXX.  Distance  to  turn  is  XXX  yds.  Next  course  is  XXX.  Atthistime, 
Recommend  (come  (Right/Left)  to  regain  track  /  maintain  course  and  speed).- out  of  the  turn 

•  "'AttimeXXX,  I  holdthe shipXXXyards  (leftof/rightof/on)track.  CurrentspeedisXXkts. 
Currentcourse  overground  is  XXX.  Distance  to  turn  is  XXX  yds.  NextcourseisXXX.  Atthis 
time.  Recommend  (come  (Right/Left)  to  regain  track  /  maintain  course  and  speed)."  -Every 
minute  in  the  turn 

•  "200  yards  till  turn"  -at  200  yards  to  turn 

•  "100  yards  till  turn" -at  100  yards  to  turn 

•  "Recommend  come  tocourse  XXX"  -at  turn 
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The  picture  below  is  a  sampleof  what  your  heads  up  display  will  look  like.  In  order  to  activate  and 
deactivate  the  display,  simply  tap  the  headset  with  your  finger.  (You  may  need  to  use  a  little  force  to 
get  it  to  work) 

Course  Over  Ground  Current  Track  Course  Set  and  Drift  (relative) 


Speed  Over  Ground 


/ 

/ 

14.hu 

‘•C 

H?1344^ 

ISgJW 

0  SUM 

r  !' 

Distance  to  Turn 


Next  Track  Course 


Visual  representation  of 
progress  on  each  track  leg 


s 


(Flashing)  <  2(X)  yds  till  turn 


(Flashing)  <  Recommend  start  turn 


Good  Luck! 
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HUD  ONLY 


Page  1  of  1 


Instructions:  (VISUAL  ONLY) 

We  will  assist  you  in  putting  on  the  head  mounted  display.  When  the  simulation  starts  you  will 
be  placed  at  the  left  center  of  the  bridge.  The  ship  will  be  moving  in  the  water  at  15  knots  heading  in 
the  direction  of  the  first  leg.  As  soon  as  you  are  ready  to  start,  simpty  start  giving  commands  to  the 
helmsman  and  try  to  drive  as  close  to  track  as  possible.  One  of  the  performance  metrics  ofthis  study  is 
distance  offtrack. 

In  this  scenario  you  will  be  given  only  the  ability  to  use  the  Heads  Up  Display: 

The  picture  below  is  a  sample  of  whatyour  head's  up  display  will  look  like.  In  order  to  activate 
and  deactivate  the  display,  simply  tap  the  headset  with  your  finger.  (You  may  need  to  use  a  little  force 
to  get  it  to  work) 


Course  Over  Ground  Current  Track  Course 


Set  and  Drift  (relative) 


'/ 


■/ 


Speed  Over  Ground 


§14.9ku 


im 


t 


Distance  to  Turn 


Next  Track  Course 


Visual  representation  of 
progress  on  each  track  ieg 


M 


(Flashing)  <  200  yds  till  turn 


(Flashing)  <  Recommend  start  turn 


Good  Luck! 
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Debriefing  Form 

Navigational  Heads  Up  Display  (NAVHUD) 

Thank  you  for  participating  in  this  study! 

While  every  effort  in  the  design  of  the  simulation  has  been  made  to  mitigate  cyber 
sickness,  there  is  a  possibility  of  you  experiencing  some  symptoms  temporarily  during  or 
after  the  study.  Symptoms  include  visual  symptoms  (eyestrains,  blurred  vision, 
headaches),  disorientation  (vertigo,  imbalance)  and  nausea  (vomiting,  dizziness).  You 
are  advised  to  refrain  from  following  actions  for  2  hours  after  the  completion  of  your 
participation  in  the  study:  riding  a  bike,  operating  heavy  machinery  or  climbing  high 
structures.  In  case  you  experience  a  mild  form  of  any  of  the  symptoms,  you  are  advised 
to  sit  down  and  rest.  In  case  of  severe  forms  of  symptoms  you  are  advised  to  seek  the 
help  of  a  medical  professional. 

If  you  are  showing  any  symptom  at  this  time  we  recommend  you  stay  and  allow  us  to 
observe  you  every  15  minutes  until  you  are  asymptomatic. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  research,  or  you 
experience  an  injury  or  have  questions  about  any  discomforts  that  you  experience  while 
taking  part  in  this  study  please  contact  the  Principal  Investigator,  Dr.  Amela  Sadagic,  (831) 
656-3819,  osadagic@nps.edu.  Questions  about  your  rights  as  a  research  subject  or  any 
other  concerns  may  be  addressed  to  the  Navy  Postgraduate  School  IRB  Chair,  Dr.  Larry 
Shattuck,  831-656-2473,  leshattu@nps.edu. 

Statement  of  Release  (participant).  I  have  read  the  information  provided  above.  I  have 
been  given  the  opportunity  to  ask  questions  after  my  participation  in  the  study,  and  all 
my  questions  have  been  answered  to  my  satisfaction.  I  certify  that  at  the  time  of  signing 
this  form  I  am  symptom  free. 


Participant's  Signature  Date 


Statement  of  Release  (researcher).  I  certify  that  at  the  time  of  signing  this  form  the 
participant  appears  to  be  symptom  free. 


Researcher's  Signature  Date 
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APPENDIX  J.  PERFORMANCE  DISTRIBUTION  DATA 


S Summary  Statistics 

^  ir.  Summary  Statistics 

^  Lr.  Summary  Statistics 

Mean 

31501779 

Mean 

17.407197 

Mean 

14802937 

Std  Dev 

32592049 

Std  Dev 

18.604584 

Std  Dev 

18.106644 

Std  Err  Mean 

0^72362 

Std  Err  Mean 

0.1453439 

Std  Err  Mean 

0.1415273 

Upper  95%  Mean 

32.40599 

Upper  95%  Mean 

17.692087 

Upper  95%  Mean 

15.080346 

Lower  95%  Mean 

31.397568 

Lower  95%  Mean 

17.122307 

Lower  95%  Mean 

14825529 

N 

16350 

N 

16385 

N 

16368 

Condition  A 

Condition  B 

Condition  C 

Distance  Off  Track 

Distance  Off  Track 

Distance  Off  Track 
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APPENDIX  K.  PERFORMANCE  DATA  NORMALITY  PLOTS 


Normal  Probability  Plot  of  Trat*  1 

NPS  Augmented  Reality  Conning  Offioer  Performance  Study  CompressedData-averagesPerTradc 

-2015-02-23  18v-75c 


Normal  Probability  Plot  of  Trade  2 

NPS  Augmented  Reality  Conning  Officer  Performance  Study  CompressedData-averagesPerTrabc 

-2015-02-23  18v-75c 
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Normal  Probability  Plot  of  Tradt  3 

NPS  Augmented  Reality  Conning  OffitEf  Performanoe  Study  CompressedData-averagesPerTradc 

-2015^2-23  18v*75c 


Normal  Probability  Plot  of  Tradt  4 

NPS  Augmented  Reality  Conning  OffitEr  Performance  Study  CompressedData-averagesPerTradt 

-20154)2-23  18V75C 
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Normal  Probability  Plot  of  Trac*  5 

NPS  Augmented  Reality  Conning  OfficEf  Performance  Study  CompressedData-averagesPerTradc 

-2015^2-23  18v*75c 


Track  5:  SW-W  =  0.6497,  p  =  0.0000 


Observed  Value 
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APPENDIX  L.  STUDY  CONDITION  EVALUTIONS 


A.  EVALUATIONS  OF  THE  NAVIGATION  REPORT  (CONDITION  A) 


1:  Not  at  ali  weli 

2: 

3:  Average 

4 

5:Ver^ 

r  Well 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Info  was  Timely 

1 

4.0% 

0 

0.0% 

8 

32.0% 

5 

20.0% 

11 

44.0% 

Info  Was  Loud  and  Clear 

0 

0.0% 

1 

4.0% 

3 

12.0% 

3 

12.0% 

18 

72.0% 

Necessary  info  was  provided 

0 

0.0% 

1 

4.0% 

5 

20.0% 

3 

12.0% 

16 

64.0% 
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B.  EVALUATIONS  OF  THE  NAVIGATION  REPORT  (CONDITION  B) 


1;  Not  at  all 

2: 

3:  Average 

4 

5:Ver^ 

r  Well 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Info  was  Timely 

0 

0.0% 

2 

8.0% 

7 

28.0% 

7 

28.0% 

9 

36.0% 

Info  Was  Loud  and  Clear 

0 

0.0% 

0 

0.0% 

4 

16.0% 

6 

24.0% 

15 

60.0% 

Necessary  info  was  provided 

0 

0.0% 

1 

4.0% 

6 

24.0% 

6 

24.0% 

12 

48.0% 

C.  EVALUTIONS  OF  THE  HUD  (CONDITION  B) 


1:  Illegible 

2: 

3:  Average 

4 

5:  Very  Legible 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Legibility  of  Font 

0 

0.0% 

1 

4.0% 

4 

16.0% 

6 

24.0% 

14 

56.0% 

Size  of  Font 

0 

0.0% 

0 

0.0% 

3 

12.0% 

8 

32.0% 

14 

56.0% 

Contrast  of  Font 

0 

0.0% 

0 

0.0% 

3 

12.0% 

9 

36.0% 

13 

52.0% 

1:  Very  Distracting 

2: 

3: 

4 

5:  Not  Distracting 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Level  of  Distraction 

0 

0.0% 

2 

8.0% 

4 

16.0% 

10 

40.0% 

9 

36.0% 

1:  Very  Difficult 

2: 

3:  Average 

4 

5:Ven 

Easy 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Ease  of  HUD  deployment 

1 

4.0% 

1 

4.0% 

4 

16.0% 

4 

16.0% 

15 

60.0% 

1:  Not  Helpful 

2: 

3:  Average 

4 

5:  Very  Helpful 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Helpfulness  of  HUD 

0 

0.0% 

0 

0.0% 

1 

4.0% 

5 

20.0% 

19 

76.0% 
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D,  EVALUTIONS  OF  THE  HUD  (CONDITION  C) 


1:  Illegible 

2: 

3:  Average 

4 

5:  Very  Legible 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Legibility  of  Font 

0 

0.0% 

0 

0.0% 

5 

20.0% 

6 

24.0% 

14 

56.0% 

Size  of  Font 

0 

0.0% 

0 

0.0% 

4 

16.0% 

9 

36.0% 

12 

48.0% 

Contrast  of  Font 

0 

0.0% 

0 

0.0% 

4 

16.0% 

9 

36.0% 

12 

48.0% 

1:  Very  Distracting 

2: 

3: 

4 

5:  Not  Distracting 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Level  of  Distraction 

0 

0.0% 

3 

12.0% 

4 

16.0% 

9 

36.0% 

9 

36.0% 

1:  Very  Difficult 

2: 

3:  Average 

4 

5:Ven 

^  Easy 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Ease  of  HUD  deployment 

0 

0.0% 

3 

12.0% 

5 

20.0% 

3 

12.0% 

14 

56.0% 

1:  Not  Helpful 

2: 

3:  Average 

4 

5:  Very  Helpful 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Helpfulness  of  HUD 

0 

0.0% 

0 

0.0% 

2 

8.0% 

9 

36.0% 

14 

56.0% 
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E. 


POST-EXPERIMENT  EVALUTIONS 


1;  inaccurate  /  Hindered 
performance 

2: 

3:  Average  /  No  Effect 

4 

5:  Accurate /Aided 
performance 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Ship  Motion  Physics 

0 

0.0% 

2 

8.0% 

9 

36.0% 

11 

44.0% 

3 

12.0% 

Visuai  Representation  of 
objects  inside  the  ship 

0 

0.0% 

2 

8.0% 

8 

32.0% 

10 

40.0% 

5 

20.0% 

Visuai  Representation  of 
objects  outside  the  ship 

0 

0.0% 

5 

20.0% 

6 

24.0% 

8 

32.0% 

6 

24.0% 

1:  Hindered 
performance 

2: 

3:  No  Effect 

4 

5:  Aided  performance 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

How  weii  did  the  simuiated 
movement  (within  the  ship's 
bridge)  ailow  you  to  perform 
you  task? 

1 

4.0% 

0 

0.0% 

12 

48.0% 

6 

24.0% 

6 

24.0% 
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